throwdbaaway

Meow! Did nobody read the license?

Finally an article that gets the maths. Following the release of DSV4 preview where 1M context can fit in single digit GB of VRAM, frontier labs pricing just became stupidly expensive. Other Chinese labs and r/LocalLLaMA also can't compete on pricing.

And since then, there has been so many articles that made it to HN front page, and all of them didn't get it. They just went on and on about tokens generation. Most HN commenters didn't get it either, find-in-page for "cach" typically yield 2~3 responses. If I had a dime for every time this happened, I could have .. paid for 1B cached input tokens?

Anyway, DeepSeek still has to come up with a frontier model, and they almost did it with DSV4 Pro 0813, which is just slightly below GLM 5.3, but 30x cheaper. Unfortunately, the massive price hike happened just 3 days later.

DSV4.1 Flash is good, but not quite the same level. Much easy to self-host though, especially for serving a team of developers. Let's see what the next one can do.

Back to the topic about cuda moat, in the slide with title "Problems with Coopmat1", the current frontier open models have absolutely no issue with:

* manual pipelining

* shared memory staging

* tiling

* bounds checking

On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.

From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.

I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.

Few problems I noticed:

* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.

* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.

Hold on.. the routed experts are in FP8 now? Previously they were in BF16. Nice, this shall cut my download time by half!

Sounds like that's what z.ai did to get GLM-5.3-Flash running on Huawei chips.

Exactly. Coding for inference is solved. CUDA is no longer a moat.

> $20k workstation, best case: $15k M5 Ultra 512GB, 36-month amortization, ~$440/mo. Runs a GLM-5.3-class model at ~30 tok/s. Saturated 24/7 it produces roughly 58M output tokens/month.

For agentic coding, ~90% of the cost comes from cached input tokens. This cost increases quadratically with the session length. If sessions go near 1M context, the number of cached input tokens can easily exceed 1B in a day.

GLM-5.3 @ $0.26/M x 1000 = $260/day

This is the math to use.

> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...

If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.

> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)

But this failure mode can still happen.

Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.

So, perhaps the lesson here is "don't use vllm at home"?

Reproduced on the CUDA stack right?

Let's say DeepSeek is being forced to use the CANN stack, and the new pricing reflects the cost when 100% of inference is done with Huawei chips. Then, I suppose we can infer that:

* CANN stack is 1.5x~2.3x less efficient in compute

* CANN stack has 6x lower inter-connect capacity

> computer chips once again become a commodity

Ascend 950 is going for $7k to $9k with mediocre looking specs. $16k for RTX Pro 6000, $6k for RTX Pro 5000. This is not looking good.

Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full precision MXFP4 quant. Anyway, they all work fine in practice.