hiramwen

Author here. I wrote this because I wanted my private inference boxes available from other devices. Today I dogfooded the guide by serving the new Qwen3.8-27B Q4 on an old 8GB Quadro P4000 + 128GB RAM box.

It uses an OpenAI-compatible API and can power private apps or agentic sessions; I’m currently using local models for creative writing and business projects. Qwen3.8 still needs some per-machine tuning and can use a lot of thinking tokens, but in one particularly extreme test it one-shot an absolutely gorgeous app after about 19 hours of thinking.

Excellent work! The chart is gorgeous.

I added a Kelly simulator to my own personal stock tracker after seeing this

This woulda been helpful for Leo and the 3% of Korea that got liquidated

Local inference without access to internet and tools is pretty safe, worst you get is a slop output, best case it solves your prompt.

This is the best take. Return on capital is diminishing from intense competition so improvements get hidden away like the gems they are.

1) Frontier labs have no incentive to give the general public their best anymore; it's instantly distilled off of them. Why not charge governments and big corps real money to use the real good stuff instead? 2) So we get distilled-off-frontier public APIs like 5.6 and Fable/Opus. And the open source labs are distilling off of those. 3) There's a lot of benchmark hacking right now among all the publicly available models, actual usability of Opus for coding is far below its benchmarks suggest. 4) But context window, cybersecurity, logical coherence, and tool usage are absolutely better on Fable and Sol. It looks to me their internal tools definitely even better and not plateauing. But we won't get to use it.

I've been waiting for a tiny model release! Kimi, Deepseek, and GLM were all monsters, the 27B 3.8 is filling a gap thats been open since Qwen 3.5

Also I love how literal Qwen 3.5 was, hopefully 3.8 is still extremely literal. The token explosions were actually helpful in debugging prompts.

You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.