kmike84

heh, I've written/contributed to a lot of open source over last couple of decades; it's not the point.

My point is that no-AI assisted development is becoming very niche; it's not a right approach if you want to get things done, and AI-assisted doesn't mean lower quality. And if you against paying evil Anthropic and OpenAI corporations you have options like local LLMs.

Well, it's their choice. If you care about the process - i.e. it's a hobby for you, or a spiritual experience, then it's totally valid to go without AI.

But I feel it's a bit crazy not to use any AI assistance to develop real code these days, if you care about outcome. You can get to result much faster, and have a higher quality solution, if you use it right.

The problem for open source is low quality slop, not AI assistance. Not sure what's the solution though.

I think it'd be mostly for engrams. It seems LLMs are moving in this direction, with 50GB in qwen 3.8 flash, and 200GB in deepseek 4.1 flash.

My $20 xeons 2696v2 and 2670 are tool old for real offloading, and sata ssd drives are likely a bit slow for ssd streaming :)

It’s making ddr3 relevant again as well. Bought 256GB DDR3 ECC for ~200 usd a couple weeks ago for my ai server (wip).

These old workstation platforms seem to be perfectly fine for hosting GPUs for local llms.

How accurate are speed estimates in the UI? I'm asking because for Qwen 3.8 (Q8) the speed numbers cited in the UI look quite poor:

  Estimated speed on your machine
  Context tokens Tokens / sec
  25 000 17
  50 000 16
  75 000 16
  262 144 12

262K number is ok, but for lower context sizes (<128K) it's about 2x slower than the numbers I'm getting from real mtplx sessions for qwen3.8 q8 (mac m5 max).

Is it a lack of optimizations, or incorrect numbers, or a benchmark artifact (e.g. something which is harder for spec decoding than usual agentic sessions)?

I think the specific issue I had was due to lack of proper support for ds4 compressed KV cache, not about KV cache quantization. It was like 50GB instead of 5GB for context, and it wasn't fixed for weeks (I haven't checked if it's fixed now - hopefully it is).

Quantization is another thing. There are so many engines launched with claims about speed, but in many cases it's optimizing specific lower-quality quants. When you have enough resources, you usually want something like W8A16 + full precision KV cache working as fast as possible, not yet another W8A8 or W4A16.

In general, it seems new models are released so fast now - engines don't always have time to really polish the implementation before the next model is released

This seems to be a good idea. However, beating llama.cpp on speed is a low bar :)

I found it to be a good baseline, but at least on Mac there was always something way faster, and/or with better memory requirements - like you said, ds4, omlx, mtplx, etc. It seems if you use local LLMs for real, there is very little reason not to use one of the more optimized engines.

3 main failure modes I observed in the engines:

* Not using best available spec decoding

* Using too much VRAM for KV cache (e.g. KV cache used to take almost nothing in ds4, but huge amount of VRAM on unsloth/llama.cpp for deepseek models)

* Degraded performance at large context sizes - benchmarks at 4K or 32K are awesome, but at realistic 100-200K it's slower than some stupid baseline

Unfortunately no, it's 200B + 552B. It's not as bad as it sounds though, because most of 552B is in 4bit natively.

Measuring quality e2e definitely makes sense.

But I think there is a bit more to this:

> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.

A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.

I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.

You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.

Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.

Still investigating it though :)

Pass. When articles keep mentioning models like DeepSeek R1, or Llama 3.1, or Qwen3 32B, it is a pretty robust indicator of AI slop. LLMs love to suggest DeepSeek R1, etc. - training data cut-off?

No person with real practical experience and real use cases will be using these ancient models as examples, when talking about local LLMs.

So, open models will be better on this benchmark, which is deserved

Heh, a good point.

> Which harness for the benchmark ?

pi, with a plugin to do web search / web fetch.

> You have previously commented on using OC/GLM. R u going to stock with it?

For personal use - probably yes, z.ai + kimi + opencode go subscriptions, with some share of local models now. For work - claude code, codex.