What is the state of using quants? For chat models, a few errors or lost intelligence may matter a little. But what is happening to tool calling in coding agents? Does it fail catastrophically after a few steps in the agent?
I am interesting if I can run it on a 24GB RTX 4090.
I am interesting if I can run it on a 24GB RTX 4090.
Also, would vllm be a good option?