jononor

Cloud compute has managed to have pretty good margins. That seems like a really similar business model to what Anthropic is doing - inferene+value-adds as-a-service. Why wouldn't they also be able to get cloud level margins or a bit higher?

Yeah performance at a given compute budget will likely continue to improve.

We are running Qwen 3.8 27B FP8 with 4x 5060 16gb at the office for a small development team. It is doing pretty well for implementing backend and frontend tasks, and is already useful. In 6 and 12 months from how I expect it to get even better. Some planning is still done with a stronger model, but even that is maybe not strictly necessary.

Above GLM 5.3, nice! That is a very capable model. I do wish that we will see some smaller "flash" models again though - running this one locally is tough (very costly).

Everything except for LNG is down a lot. LNG seems to be basically at the same level. Does anyone know why this is?

Both Google and Meta have done that years ago, with Tensorflow/JAX and PyTorch?

But a software abstraction only helps to reduce lock in. It does not unlock efficiencies on itself. For that new and/or hardware hardware might be needed.

The default state of agents and LLMs is inert. It requires action from a human even be able to do something. From the very basics like starting the software, connecting it to a network, having hardware to run on.

But the most important wrt AI is to keep the owners/operators responsible. Don't let them weasel their way out of it. They are for sure trying, and will continue to. This includes using language to overemphasise agents importance in bad outcomes, in order to downplay their own responsibility.

Agreed. AI as an "accountability sink" is an incredibly bad idea. It allows/incentives bad actors to do bad shit and get away with it. Which will, generally, tend in such practice becoming more common. Everyone loses except for the crooks. We cannot accept "AI" absolving humans of responsibility.

Might even be cheaper!

Some kind of compute-in-memory architecture is a good candidate, I think. There are many alternatives here, researched for many years prior to the LLM craze. However economies of scale dominate in chip industries, and this tends to favor more conventional or incremental approaches (to piggyback on existing scale). Alternatively someone needs to have a way of bootstrapping the insane scales needed to be competitive with a better-but-different approach. So it could be that boring and straightforward stuff like two-chip prefill+decode takes most.

I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High Bandwidth Flash (HBF) are relevant here, and should come to market in a few years. Further optimization may be possible by tailoring for sequential access, since inference of a particular model is very predictable.

That is not great. But it is still better than other continents?

If you have good feedback signals, like tests/benchmarks/etc, then it is potentially better to do multiple turns where model uses that to adjust code. Which might not need as smart a model.

If Simon would pitch for example PCBWay that and I am pretty sure they will sponsor it (assuming their logo stays). They can do laser engraved versions also ;)

It depends on your tolerance level for having less than frontier LLM capabilities. First level worth trying, Qwen 3.6 35B A3B with a 16 GB VRAM (example 1x 5060ti 16gb, 600 USD for the card) with partial GPU offloading. Next level would be Qwen 3.6 27B / new Muse Spark / Gemma 31B with 32 GB VRAM (2x 5060ti or 1x 9700 Pro). Third level would be DeepSeek V4 Flash with 192 GB VRAM (2x Strix Halo at some 8000 USD total). These models can be tried on OpenRuouter etc, or you can deploy vLLM on rented GPUs to get a feel for what level you would want before committing to buying hardware.

AI seems to be on the same trajectory? Search was very useful in the start also, until it became entrenched. Then search placement became a target, and they are just focusing on extracting rents. All way paying the content providers zero or near-zero. And with years of that dynamic, we end up where we are now. It was the same with "social media". The same will happen with AI. AI is a power for more enshittification - being currently less shit than Google is (mosy likely) temporary.