A few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my own computer. I think the M2 Pro has 200GB/sec internal bandwidth, about 25% faster than the M3.
I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.
I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.