For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.