Comment

Comment by eyegor

Oh you should be running flash next with strata. Just go to stratas github page [0] and it will explain everything. It runs quite fast on 16+64, you should be able to get 50 tok/s or so. It's a 180b model and it feels a lot smarter than most of the other open weight options. As far as knowing "what will fit" in general that's hard to know these days. The moe models are hard to predict and run wildly different in different engines (llama.cpp, vllm, strata likes, etc) but they tend to be the only good options around 16gb vram. Dense models like qwen3.8 27b are only good if they're entirely in vram. Moe models you need the active count + a few experts (although it will run faster the more experts you can fit).

[0] https://github.com/Niko1221/Strata

parent eyegor 16 days ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton