davidetroiani

So I built a runtime based on llama.cpp that uses SSD offloading and Lookahead Expert Scheduling. I introduced optimizations like adaptive caching and prefill batching, I/O and GPU work overlap, GPU keepalive to keep frequency high. Turns out that this runtime now lets me run Qwen 3.8 Flash Next (RCO GSQ IQ3_XXS from ISTA DASLab) at 10 t/s decoding, 160 t/s in prefill and keep it 7.5GB peak process memory footprint. That’s double the end-to-end speed and half the RAM footprint compared to Strata with the same model on this machine.

Currently the project is an open-source alpha, I am still optimizing and adding support to more model architectures. Currently tested models are Qwen 3.8 FN and Qwen 3.6 35B A3B (Unsloth’s Q6) that runs at 400+ t/s in prefill.

The project is called MoEShift and I’d love to see how it performs on your Macs: you can share the benchmark as a GitHub issue using the benchmark runner that I prepared. Of course technical feedback is much appreciated.