I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.