bitexploder

I have ended up using Argon more and more. If you keep it grounded it is a great model. Extremely thorough. A little slow. It feels like it gets things done better than Opus 5.5. It slowly churns through everything. Not necessarily quality but its adherence to goals seems better and it takes less prompting.

For cyber use cases I am absolutely blown away. Can’t say much more, but it is a serious model.

Just my opinions. I really recommend people take it seriously. For many jobs I just prefer it now. It is a new model so it needs time. Opus 5.5 is a sweet spot as well. I use both extensively.

With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.

Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)

You will end up with a sub-par result. The space for small models is more specialized and specific and you should optimize to the specific task and or tasks you want to do. It will get some local model working, but there are a lot of interesting small models in the 4-10B space to look at and play with for your specific tasks. If you expect a full agentic coding agent it's just not going to happen without a larger model like Qwen 35B and ... that is not that great by today's standards, even Muse Glimmer is better.

Local AI already is really useful. This has been true since Qwen Flash Next and Qwen 3.8 27B rolled out. The struggle with exactly 64GB of unified RAM is that you end up having to stream the engram table from SSD because you will need almost all of your 64GB to run QFN. A 27B dense model still runs a little slow, even on an M1 Max Mac. However if you have even an older gaming GPU with 12-16GB of RAM and enough system RAM you can run this model (QFN) at a very respectable token rate and intelligence level. It is a Sonnet 5 class model you can run at home. There are new quantization techniques that make these models very good at agentic tasks.

And now if you stretch you can run DeepSeek Flash 4.1.

And if you were willing to roll up your sleeves you could build your own AI lab at home and run this model and models like GLM Flash 5.3 and DeepSeek Flash V4.1 at lower quants. The budget GPUs (like V100) are kind of not budget now, but still <5K to acquire 128GB of fast VRAM that can compete with a 5090 and actually has a lot more interesting options available IMO.

Waiting for 512 GB Mac Studio M5 Ultra still. I guess they are rolling things out now, it has been expected for late October.

A long time ago, in a galaxy far away, I used to write a bunch of ColdFusion with a dude that owned an S2000 :)

It used by be a markup language like:

  <cfif blah="something is true">do this<cfelse>do that</cfif>

But now it has its own scripting language like:

  <cfscript>
  // scripty stuff
  </cfscript>

At least, that is how it used to be :)

I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.

Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.

And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)

I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.

My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.

You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.

I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.

Not if you have your own TB of fast RAM.

The math. It’s pretty brutal. You have to multiply insanely large matrices in memory. A dense model has to have and multiply across every parameter basically. MoE models work around it, sure, but you still have to hold a lot in memory and density of activated parameters matters. There is not much of a way around it yet. We are potentially getting there, but its not obvious yet.

I have 27B at around 160-170 t/s on my hardware but why bother when QFN is faster and generally better :)