pistoriusp

Same, that percentage seems absurd though.

> Since any LLM will answer all those questions competently

That's false. The LLM will only answer competently if it was trained on that data; and if it has enough data to make the correct connections between your question and the "correct" answer.

In the case of this article they're specifically saying the LLM has limited training.

Aren't you missing OP's point entirely? Which is: If the LLM didn't have useful information it would still give you an answer... Helpful or not.

Which is why you should perform your own benchmarks against your own software stack.

I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.

It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

I think this is less about the actual use-case and more about the approach; so a comparison is moot.

Interesting. I had no idea that a person could see what is flagged?

I had the same experience, but someone calmed me down by saying "we're all working on solving this problem."

You've got to place that within the context of Sub-Saharan Africa.

Mac Mini in the rack and a Neo in the lap.

Why not just run a vm?