I'm surprised that no media outlet is seriously talking about its on-device inference capability, which is the biggest next thing for LLM. Apple did a better job on its PR here, but they didn't also get all the attention that it deserves.
On-device inference improves not just its latency, this also removes a huge chunk of LLM's economical constraints from software companies. The biggest advantage of software is its marginal cost being nearly zero. LLM hasn't enjoyed this luxuries but the dynamic is going to change.
On-device inference improves not just its latency, this also removes a huge chunk of LLM's economical constraints from software companies. The biggest advantage of software is its marginal cost being nearly zero. LLM hasn't enjoyed this luxuries but the dynamic is going to change.