Absolutely! Even for inference! The SOTA models for all commercial purposes need to run on a consumer’s device.
Running either Grok2 or DeepSeek or even Llama405b requires nearly 400-500gb of memory.
Buying a tinybox with enough gpu memory costs $15k-25k. Or equivalently the same if you build your own.
A distributed Mac cluster costs about the same, if not more, if you’re buying 2-3 M2 Ultra each with 192gb of memory.
So people are absolutely constrained by price/supply here. Every engineer, analyst, scientist would be far more untethered by rules & regulations or policies & terms-of-service nitty gritties if they can trust that LLM they use is completely local, without-telemetry or tracking and is licensed fairly for commercial use (perhaps this excludes llama).
Not a lot of people can afford $15k-30k in spending for a computer (that can run this sota llms). But you can a billion will buy one when it’s $1k
Running either Grok2 or DeepSeek or even Llama405b requires nearly 400-500gb of memory.
Buying a tinybox with enough gpu memory costs $15k-25k. Or equivalently the same if you build your own.
A distributed Mac cluster costs about the same, if not more, if you’re buying 2-3 M2 Ultra each with 192gb of memory.
So people are absolutely constrained by price/supply here. Every engineer, analyst, scientist would be far more untethered by rules & regulations or policies & terms-of-service nitty gritties if they can trust that LLM they use is completely local, without-telemetry or tracking and is licensed fairly for commercial use (perhaps this excludes llama).
Not a lot of people can afford $15k-30k in spending for a computer (that can run this sota llms). But you can a billion will buy one when it’s $1k