I just checked and MLC Chat is running the 3-bit quantized version of Mistral-7B. It works fine on the 14 Pro Max (6GB RAM) without crashing, and is able to stay resident in memory on the 15 Pro Max (8GB RAM) when switching with another not-too-heavy app. 2-bit quantization just feels like a step too far, but I’ll give it a try.
Regarding credit, I definitely don’t need any. Just happy to see someone working on a better LLM app!
Regarding credit, I definitely don’t need any. Just happy to see someone working on a better LLM app!