bisonbear

IMO for an involved voter, this is actually a good thing. AI can both help you clearly define your priorities and policy preferences via an "interview", and can subsequently find candidates who's publicly stated positions, and past record, align most closely with your priorities.

As always, AI can be used to completely outsource critical thinking, and doing so for elections is extremely concerning. However I would hope that more people use it to make better, more informed decisions, than to blindly choose candidates

Pretty cool to hear about your process - I've been thinking about this space a lot so have some context about pain points here.

I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.

For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?

I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!

Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.

Sharing some challenges / learnings from the journey so far:

* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result

* However, tests are still important and the best form of deterministic evaluation we have

* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos

* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change

* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading

* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.

The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.

I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).

Also working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winning" means. If just using test pass rate, consider that tests might not encode what good means in your repo. I have been having success using "equivalence with merged PR" as judged by an LLM as a signal.