spmurrayzzz

My expectation is always that sm120 is the forgotten orphan in the mix and that I'll have to monkey-patch my way to getting things running (and its not always due to lack of tcgen05 or other "real blackwell" features, its often just failing fallback kernel paths doing things that were clearly not tested).

Mojo's design goals would in theory help this problem long term, but my sm120 PTSD still lingers.

MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all.

THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).

When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.

Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).

Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.

4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.

> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.

I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.

The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.

(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)

Slop language aside, the abstract insight is directionally correct I think.

CPUs were already becoming much more important during training for test-time scaling, but there you were still bottlenecked by GPU compute since the gradient updates back to the policy model are the actual gating factor.

During normal inference though, CPUs are becoming more of a bottleneck for more advanced workloads. Even if you have 20 agents running in parallel, if they're all compiling Rust concurrently your total wall-clock time per task is no longer bound by the decode throughput of the upstream model. You're just waiting for tools to execute. This gets compounded by VM/container overhead as well if you're doing the totally local sandbox approach.

Given the current throughput figures on OpenRouter (~180 tk/s), its likely a much smaller param count on the order of something like Luna. I think the better, more timely comparison (re: your point on Chinese labs) would be to DeepSeek-V4-Flash-0731.

It's definitely confusing from a presentation perspective, but they are somewhat coherent comparisons if you account for the inference heuristics involved.

(They could in theory be gaming the decode speeds with much larger than normal batch sizes given the TTFT is pretty high at around 8s)

When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.

And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).

I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work).

Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.

I had a similar reaction to the "I work at an AI company" and finding out it was Dropbox. And I agree with you, they are not in any way an AI company that would be relevant for someone making claims about frontier intelligence.

I'm empathetic to their position though. It is entirely unsurprising to me that someone working in a growth role at Dropbox is unimpressed by the current state of AI relative to its broader claims in the market. They're not working on AI itself nor are they using applied AI where you see the biggest gains (e.g. SWE, ML, data science, etc.).

We still have a significant capability overhang at the frontier for a big chunk of knowledge work task domains, so I think its understandable (given the above selection bias) why someone would think the confidence is overblown. They have a point in their own domain.

I'm not entirely up to date with the latest batch, but I've reviewed some of the rollouts in the past and my sense is that the models are surprisingly good at getting correct custom kernels in the happy path, but still weak at sustained/shape-robust workloads. Having to deal with writing the full path from scratch compounded by weird memory layouts, odd sizes, routing, unpacking quantized weights, etc. is definitely challenging.

Also, at least a portion of this you could argue is arbitrary and entirely scoped to the eval itself. The fp8 GEMM score could be low simply because one of the shapes is fairly skinny (i.e. not enough math work to keep the compute engine busy for a meaningful amount of time).