RyeCatcher

Hey HN, author here. Happy to answer questions.

Why Rust? - Ownership model makes the CoW block table provably safe, the borrow checker enforces that you can't alias a block that's being written. That's not just style; it eliminates a whole class of correctness bugs that plague Python/C++ inference engines.

How is this different from vLLM's paged attention? - vLLM pages memory in fixed blocks to avoid fragmentation. Dendrite does that too, but adds O(1) KV cache forking via copy-on-write. When you branch a beam or MCTS node, you get a shallow pointer copy (~500ns) instead of copying the full KV cache. The deeper the tree, the bigger the win.

TurboQuant: Google published this last week (ICLR 2026). We already have a Rust implementation, PolarQuant + QJL pipeline in `cache/compress.rs`. Measured 3x compression at head_dim=128 on CPU; paper claims 6x with per-head grouping (coming).

Status: This is research-grade, not production. No Python bindings yet (tracked in issues), no FlashAttention kernels. Best fit today: if you're building tree-structured search (MCTS, beam, speculative decoding) and want to control the stack.

Author here. I've updated the article based on your feedback. Thank you.

Key corrections:

Ollama GPU usage - I was wrong. It IS using GPU (verified 96% utilization). My "CPU-optimized backend" claim was incorrect.

FP16 vs BF16 - enum caught the critical gap: I trained with BF16, tested inference with FP16 (broken), but never tested BF16 inference. "GPU inference fundamentally broken" was overclaimed. Should be "FP16 has issues, BF16 untested (likely works)."

llama.cpp - veber-alex's official benchmark link proves it works. My issues were likely version-specific, not representative.

ARM64+CUDA maturity - bradfa was right about Jetson history. ARM64+CUDA is mature. The new combination is Blackwell+ARM64, not ARM64+CUDA itself.

The HN community caught my incomplete testing, overclaimed conclusions, and factual errors.

Ship early, iterate publicly, accept criticism gracefully.

Thanks especially to enum, veber-alex, bradfa, furyofantares, stuckinhell, jasonjmcghee, eadwu, and renaudr. The article is significantly better now.

Cool I’ll have a look. All reflections I made were first pass stuff.

Need to buy 2 and connect em. :-)

Totally agree. I’ve been training nanochat models all morning. Hit some speed bumps. I’ll share more later in another article. Buts it’s absolutely amazing. I fine tuned a Gemma3 model in a day yesterday.

I absolutely love it. I’ve been up for days playing with it. But there are some bleeding edge issues. I tried to write a balanced article. I would highly recommend for people that love to get their hands dirty. Blows away any consumer GPU.

Would love to hear from others using the spark for model training and development.