I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing
And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.
And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.