pcwelder

Epistemic trespassing

We just need transparency. It's okay if you change criteria, but pre-register the trigger for when you're going to change them.

Example could be, we'll update X criteria when top-N models are within 2 points of each other.

Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.

Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.

Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.

In our own benchmark we found that K3.1 pro UI designs are very similar to OpenAI's designs.

K3 for reference had 0.43 similarity to OpenAI's models, now it's as similar as how OpenAI's models are to each other at 0.58, much higher than other cross-lab similarities.

Biggest gripe with K3 was slow responses, partly due to inefficient thinking. So far K3.1 pro is almost twice as fast as k3 on our benchmark.

Is there a mention of context length? I can't find it. I could not integrate Jev due to limited window (64k?).

The whole thing (article, benchmark) is a slop soup.

- Metaphor overload. We get it, it's just like car racing. Show some mercy on human readers.

- X, not Y

- A, never B

- Tasteless em dashes

- Hallucinated data, like model add date. The standings are so unbelievable that they border on laughable.

The most I enjoy working with AI is my special workflow.

I ask it to plan the feature in a separate worktree.

In parallel I start coding without being biased by AI and vice versa.

At some point I read its plan and iterate on it all the while I am in implementation mode. This helps me improve my own vision.

Finally I ask the AI to review my implementation. It flags off bugs and gaps which are usually straightforward for it to fix.

Could be json or non json. Instead of using tools in API, you ask model to share structured output in text. You parse the string to get the JSON. Gives much more control over things you can do.

For example model shares

<tool_call name="getWeather"> <param name="city">London</param> </tool_call>

Great! Finally follows custom tool call format (k2.6 couldn't). It's a good indicator of instructions following and agentic behaviour.

UIs it's generating is pretty good, not without problems, but certainly better than other models at this price point.