> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?