The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.
I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.