It's not a new technique at this point, but it still blows my mind that you can quantize a model trained at fp16 down to 4 bits, and still have coherent inference results.
> While running, the model uses about 4GB of RAM and Activity Monitor shows it using 748% CPU - which makes sense since I told it to use 8 CPU cores.
> I imagine it's possible to run a larger model such as 13B on this hardware, but I've not figured out how to do that yet.
Naively it seems like you could repeat the same process outlined in the article but with "13B" in place of "7B". What's the catch?
> While running, the model uses about 4GB of RAM and Activity Monitor shows it using 748% CPU - which makes sense since I told it to use 8 CPU cores.
> I imagine it's possible to run a larger model such as 13B on this hardware, but I've not figured out how to do that yet.
Naively it seems like you could repeat the same process outlined in the article but with "13B" in place of "7B". What's the catch?