Kimi K3 has been demonstrated on a top-of-the-line laptop Mac running at several seconds per token (it has to fetch essentially all experts from SSD, since IIRC the dense part of the model takes around ~105GB on its own!). That's impressive enough if you only care about pure Q&A/chat or pure unattended tasks, but it's not what one would expect for agentic work. You'd need quite a bit of additional RAM to have good caching for the most common experts, and even then that would only help so much. MiMo Pro would be a roughly similar story.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.