cafkafk

Hey, author here, not sure how popular this article is gonna be with HN, but want it to get out there.

This mostly just establishes some words that I'm using when discussing NixOS and the wider ecosystem.

Formalizing them hopefully will make it easier for others to reason and argue about functional system distributions without getting confused and talking past each other.

And with the new trademark, having ways to say "a thing like nix" that don't get you in trouble with the trademark seemed useful.

I hope this also make people think about how they can make things like Nix(OS), not just fork them, and that this is a general paradigm and not just something NixOS and GuixSD gets to do.

Author here. This came out of reading Denmark's national biological threat assessment (first update since 2020), which splits its AI finding in two: not much help for people without a professional background, meaningful help for people who already have one. I suspected that it wouldn't reach as far on its own, so I've tried to get some of its ideas across, as well as some of my own.

I think it's important now that some frontier models approach to making biology guardrails is near a blanket ban. The literature doesn't necessarily support that approach, and I think it's a second order consequence of how models are trained to be unbiased, rather than e.g. taking more opinionated stances.

Another point is that if automation actually displaces people in pharma and biotech research, those are significantly more likely threat actors, meaning that managing the social consequences of large language models for critical sectors is directly a part of ensuring they don't have actual "doomsday" like outcomes.

Most alignment discussions I've seen focus on cybersecurity, and neglects e.g. the reality of counterterrorism, social impacts that lead to bad outcomes, and so on.

My hope is we can see frontier labs adopt a more sophisticated approach to guardrails, and change the discourse to be more holistic and grounded in the real risks.

That "I actually don't have any personal criticisms of Jarred" made me do a spit take, because a majority of what I had just read was absolutely a personally targeted criticism of Jarred.

Whether that's "okay" is a totally separate issue. I have no idea what the history here is, or whether this is warranted. But that was absolutely a personal criticism!

Hi HN. Follow-up to the Xeon post from a couple of weeks ago. A lot of people came away from that one with a 25-flag command and no real idea which flags materially did anything, and the honest truth is neither did I fully. So I went back and measured it, one flag off at a time, 174 server restarts, with the engine log as the source of truth instead of my own assumptions.

This also is an attempt at starting to answer the "cool, but do you have numers for this" question, which is harder than one would assume!

Two things I'll flag up front, because they're corrections to my own post: --spec-autotune, which I'd called the way to squeeze the most out of speculation, is actually the worst speculation setting I tested. Ouch.

And --mla-use, which I presented as active, isn't even wired for Gemma 4 and gets silently ignored.

More broadly, most of the config does nothing for a typical setup if you're not "holding it right". The flags that genuinely carry it (flash attention, the physical-core thread count, a fixed draft length) are a much shorter list than the command suggests, and the drafter turns out to be a win for code but a loss for summarization.

And to be clear. The numbers are specific to this box, and none of it runs without the ik_llama.cpp fork I link. I'm still not an ML engineer, so if I've gotten something wrong I'd honestly rather hear it, that's the best kind of reply.

The box is now back to being busy as a Nix cache once again now, so answers are best effort.

(There's a... fifth(?!) post coming on benchmarking quantization, where the single fastest config I measured turned out to be pure garbage. But making it not garbage was useful. No spoilers!)

Loading will take some minutes, but at 96 you can squeeze the model in and have some headroom around like ~10 GB, although depending on the Xeon, you may have to downgrade to E4B instead. Should still work thou.

If you get the inference engine to route the heavy matrix math to the GPU and the speculative drafting to the CPU without choking on latency it's probably gonna be very fast.

Would love to see the benchmarks if someone actually pulls something like that off.

That is a very fair point! I just ran a not very scientific benchmark with the system under load, and posted the raw logs in a sibling comment above, but the short answer is that it's hitting 11.94 tokens per second for generation - while it's also being a binary cache and CI build server.

Totally just vibes based, I think it goes up to 20+ tps when it's not under load (and that's me trying to be conservative). For context, reading speed at 250 wpm would be around 5 to 6 tokens per second.

Honestly, at this point you're probably looking at a smaller model, for the Gemma series I'd go with Gemma 4 E4B with drafters, but that's just a hunch from using it on my laptop (where I do have a RTX 4060 M and 96gb ram).

So you'd change the invocation slightly here, but a lot of things you can potentially reuse.

That said, the Gemma 4 E4B models have so far in my experience been... not great when it comes to long context, but they are very passable for basic tasks, and even seem surprisingly okay at tool calls.

> (purple on black is really hard to read)

Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes.

> You say it runs "at reading speed". Have you benchmarked it?

At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now the system is noisy, but that said doing it like this:

llama-cli --model gemma-4-26B-A4B-it-Q8_0.gguf --model-draft gemma-4-26B-A4B-t-assistant-GGUF/wikitext-2-raw_ik-llama-mtp_drafter-conservative/gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --color -sm graph -smgs -sas -mea 256 --split-mode-f32 --temp 0.7 --cpu-moe -t 8 --flash-attn on --mla-use 3 --merge-up-gate-experts --special --mlock --run-time-repack --spec-autotune --no-kv-offload --parallel 8 --jinja -p "Why is the sky blue?" -n 128

Gives:

  llama_print_timings:        load time =   83911.65 ms
  llama_print_timings:      sample time =      26.99 ms /   128 runs   (    0.21 ms per token,  4742.15 tokens per second)
  llama_print_timings: prompt eval time =     343.41 ms /     7 tokens (   49.06 ms per token,    20.38 tokens per second)
  llama_print_timings:        eval time =   10639.36 ms /   127 runs   (   83.77 ms per token,    11.94 tokens per second)
  llama_print_timings:       total time =   11114.98 ms /   134 tokens

So 11.94 tokens per second while it's also playing binary cache and CI builder.

When I do it properly, I'll add it to the blog as well!

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers.

I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.

I've also linked the quants at the end, but they're not gonna run unless you use the ik_llama-cpp fork I mention, see other posts for more details.

I'm not an ML engineer, so I'm by no means an expert, and the server is busy acting as a Nix cache, but if you have any question, I can try to answer, but best effort.