ttsiodras
Joined 16 years ago
Karma
1,693
Submissions
195
Member since
Mar 16
Recent submissions
Comment · 1 month ago
You read that right. From the article:
"Affected products: ... Up to (excluding) 152.0.7977.82"
.82 is fixed.
Comment · 4 months ago
Just chiming in - the claims above are real, I have very similar numbers in a cluster of 2x GX10 I have access to.
Instructions to reproduce, and benchmarks here: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...
Comment · 4 months ago
I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard).
But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability.
Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.
Comment · 4 months ago
Interesting article.
IMHO, the author could have done two things better:
- vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh".
- The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork).
If you're wondering about how I set it up in the 2 GX10s: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...
Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).
Comment · 5 months ago
Thanks, Mark - for both setups!
It's very nice seeing it put to use in actual Spectrum machines - love it :-)
Comment · 5 months ago
Oh wow! I never expected to see this in a real Speccy until I fix the one I have back home (I'm an expat).
Thank you so much! :-)
Comment · 11 months ago
Thank you!
Comment · 1 year ago
I have been using xterm's default dabbrev-expand to do the same via "Alt-/" ( https://github.com/ttsiodras/dotfiles/blob/master/.Xresource... ) - which works regardless of what shell you're in.
But I was curious for your approach... so I asked Claude to convert it to bash: https://claude.ai/public/artifacts/01a49347-1617-4afe-8476-0...
Works like a charm - pinned it to Ctrl-k, which was free in my setup. I guess I don't have to depend on XTerm for this any more :-)
Thanks!
web: https://www.thanassis.space
CV: https://www.thanassis.space/cv.pdf