lihanc111

Yeah surprisingly the text of this blog is almost entirely generated by AI :) I gave all the ideas though

Actually not, i think the --- was just mistakenly typed XD

Hey HN,

We dug into 17 billion tokens of behavioral data across 413K AI agent trajectories (CoderForge-Preview) attempting real GitHub issues. Instead of just looking at final SWE-bench scores, we compared successful runs against failing runs on the exact same problem to filter out task-difficulty confounds.

The biggest surprise? Agents are not junior developers, and prompting them to act like humans actively hurts their success rate.

Here is what the data actually shows:

Human exploration rituals predict failure: "View-before-edit" and "grep-before-edit" are negatively correlated with success. Humans do this to build mental models. Agents already have the codebase in their context window; if they are heavily grepping, they aren't learning, they're flailing.

TDD is the ultimate predictor of success: The single strongest behavioral signal of a passing agent is the fraction of early bash commands dedicated exclusively to running the test suite.

The Single Responsibility Principle is law: Agents that scatter edits across 3 or more files in the first 30% of their run see their success rate plummet. Successful agents fix one targeted thing at a time.

Perseverance is a myth: If an agent runs the exact same bash command twice early on, it’s a massive failure signal. They don't adapt; they just get stuck in a loop.

Check out the article for full content!

Evolving agent systems (agents that use continual learning, memory, or context refinement) are becoming the mainstream. However, up until now, most papers show gains on only about 10 tasks, or small subsets of existing benchmarks.

The bigger issue is benchmark saturation. We highlight "circle packing" in the post as a prime example: four different recent papers (ThetaEvolve, TTT-Discover, AdaEvolve, and EvoX) all converge to the exact same ceiling (2.635983). Once multiple methods collapse to the same endpoint because they are essentially just wrapping the same SLSQP Python script, the benchmark stops measuring open-ended improvement.

To fix the evaluation gap, we built Frontier-CS. It consists of 172 algorithmic tasks and 68 research tasks, designed by ICPC World Finalists and CS PhDs.

Our main goal was to create environments with massive search spaces that require actual information-efficient exploration and long-horizon reasoning (like our hidden Permutation task), while still keeping the scoring 100% deterministic so we don't have to rely on "LLM-as-a-judge." SkyDiscover is already using this for 200+ task benchmark suites to get much cleaner, less noisy data on agent performance.

We created a benchmark for open-ended computer science problems including algorithmic problems handpicked by former IOI gold medalists and research problems in various areas by PhD students. Read more in the blog and leaderboard for how frontier models and popular evolve frameworks perform.

Please send to [email protected]

It is almost true for both. Although for the second case you can just skip storing in these cases where there is little improvement.

It is in IBM's llm-d open source stack