Comment

Comment by Doxin

I've got qwen3.6-something-or-other (also around 20GB) in 12GB VRAM with a 64k context, and I'm decently sure I could go to a 128k context without losing too much speed. it's doing around 40 tokens per second, though the initial prompt processing is a bit of a bottleneck.

MoE models are kind of bonkers.

parent Doxin 25 days ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton