There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.
I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.
There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.
I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.