Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago.
I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.
The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.
That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.
Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.
I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.
The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.
That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.
Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.