ak_t
Joined 3 months ago
Karma
102
Submissions
9
Member since
Jul 12
Recent submissions
Comment · 2 months ago
Many benchmarks now measure the total cost or energy usage per completed task.
Comment · 2 months ago
You don't have to send every single request twice, just the ones that are haven't returned in time. Wait until some threshold, such as your p95 latency, and send your backup request after that. Return whichever request comes back first, and it should cut your tail latency without doubling your cost, since it only duplicates the small % of requests at the tail.
Google calls this a 'hedged request': https://cacm.acm.org/research/the-tail-at-scale/
Comment · 2 months ago
It's been pretty consistent, the smaller models (hundreds of billions of params) usually catch up in 6 months or so to their frontier counterparts, at least on benchmarks.
Comment · 2 months ago
I think it is more likely that a smaller model (<400B) with similar intelligence gets developed long before the hardware to serve a 2.4T model gets cheaper than 10k.
Comment · 2 months ago
It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.
Comment · 2 months ago
Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago.
I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.
The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.
That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.
Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.
Comment · 2 months ago
I haven't live there since 2020, but before that it was all various SaaS and data pipeline related billboards.
Comment · 2 months ago
It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.