That MoE model is fast. 30B A3B gets 140 tokens/second on my M3 Max, and…the A3B part means a reasonable KV cache size and reasonable prefill times. 3.8 27B is a dense model that is mostly unusable, 40 tokens/second using MTP sure, but any cache miss is a few minutes of prefill. You are better off going with a lobotomized 3.8 flash MoE for the same amount of memory.
The report is still bizarre, but that model is still very relevant to local users.
The report is still bizarre, but that model is still very relevant to local users.