fotcorn

Intel really doesn't have a great product range for AI, at least compared to NVIDIA and AMD.

Their server CPUs are quite competitive and relevant for agentic stuff, but AMD CPUs are arguably better and every hyperscaler (and NVIDIA) has their own ARM based CPU nowadays.

On the GPU/AI accelerator side, they have basically nothing. Gaudi is dead, all the Ponte * accelerators are heavily delayed, Xe based GPUs are not competitive.

On client (think gaming and office PCs), memory has gotten so expensive that nobody buys new machines right now.

Just to be clear, I think this is a very small disadvantage. GCC and therefore C/C++ supports some old stuff like SuperH, Intel Itanium, PA-RISC and a bunch of microcontroller archs that LLVM does not.

However, there is now a Rust codegen plugin for GCC, so even this disadvantage is now basically moot.

Rewrite all the things!

> fixing corrupted input handling in a C/C++ code base would not be too hard

The best programmers on the planet have tried and failed with this task for 50 years now, so I don't think this is true.

The main disadvantage of Rust right now is not supporting some more obscure platforms, but because mold wouldn't support them anyway I don't see that as a problem.

I am also running some old AMD datacenter cards, 2x MI25 in my case. Getting around 30 tokens/second with short context.

Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins.

ROCm nightlies do actually support these old cards, just not the ROCm stable releases.

Also, there is zero reason to think that the big labs did not have anything similar to TurboQuant for a long time already.

The recent blog post from Google announcing TurboQuant does not change anything regarding RAM planning for the big labs.

TurboQuant itself is already a year old! So even smaller labs have probably seen and implemented it.

Cheapest i know if is around $96k

The memory bandwith on M4 Max is 546 GB/s, M5 Max is 614GB/s, so not a huge jump.

The new tensor cores, sorry, "Neural Accelerator" only really help with prompt preprocessing aka prefill, and not with token generation. Token generation is memory bound.

Hopefully the Ultra version (if it exists) has a bigger jump in memory bandwidth and maximum RAM.

Related to this, how do you get your comments that you add in the review back into your agent (Claude Code, Cursor, Codex etc.)? Everybody talks about AI doing the code review, but I want a solution for the inverse - I review AI code and it should then go away and fix all the comments, and then update the PR.

I think the ability to actually run the code on the target helped a lot with understanding and optimizing for the specific micro architecture. Quite a few of the ideas turned out to not to be optimal and were discarded.

Also important to have a few test cases the agent can quickly check against, it will often generate wrong code, but if that is easily detectable the agent can fix it and continue quickly.

I used Claude Opus 4.5 inside Cursor to write RISC-V Vector/SIMD code. Specifically Depthwise Convolution and normal Convolution layers for a CNN.

I started out by letting it write a naive C version without intrinsic, and validated it against the PyTorch version.

Then I asked it (and two other models, Gemini 3.0 and GPT 5.1) to come up with some ideas on how to make it faster using SIMD vector instructions and write those down as markdown files.

Finally, I started the agent loop by giving Cursor those three markdown files, the naive C code and some more information on how to compile the code, and also an SSH command where it can upload the program and test it.

It then tested a few different variants, ran it on the target (RISC-V SBC, OrangePI RV2) to check if it improves runtime, and then continue from there. It did this 10 times, until it arrived at the final version.

The final code is very readable, and faster than any other library or compiler that I have found so far. I think the clear guardrails (output has to match exactly the reference output from PyTorch, performance must be better than before) makes this work very well.

I have the PCIe version of NanoKVM, and I am also happy with it.

The big advantage of the PCIe version is that it does not take up space on the desk and all the cables for ATX power control an inside the PC case.

Full-sized HDMI is nice, the only limitation here is 1080p resolution. 1440p or higher would allow mirroring the output on the main monitor to the NanoKVM, but this probably a weird use-case anyway.