itamarcode

Specifically for generating unit regression tests the Cover-Agent tool already works quite well in the wild for some projects, especially isolated projects (as opposed to complex enterprise-level code). You can see in the few (somewhat cherry-picked) examples we posted [0] that it generates working tests that increase coverage (they were cherry-picked in the sense that these are examples we like to work with often internally at CodiumAI).

I believe that it’s possible to generate additional meaningful tests including end-to-end tests by creating a more sophisticated flow that uses prompting techniques like reflection on the code and existing tests, and generates the tests iteratively, feeding errors and failures back to the LLM to let it fix them. Just as an example. This is somewhat similar to the approach we used with AlphaCodium [1] which hit 54% on the CodeContests benchmark (DeepMind’s AlphaCode 2 hit 43% [2] with the equivalent amount of LLM calls).

We’ve come up with a bit of a different concept for what a coding agent should be. We believe it should work in tandem with a developer inside the IDE. Over time as the tech improves, it will get more and more autonomy.

We’ve been using our coding agent internally and see a 5-10x boost on some tasks.

The agent is available now to Codiumate VS Code users. We want to hear what kind of tasks it works well on and improve it over time to expand the task set. Would love to get feedback.