iopapa

Results for GPT-6 Astra and Gemini Flash 3.8 are just in! GPT-6 claimed the first spot with a score of 69.3. Gemini 3.8 Flash on a very solid 5th spot with 55.4.

We run each model multiple times against each challenge and take the average score. We include the variance below the score in the leaderboard.

GPT 5.5: 42.3±10.1 GPT 5.6 sol: 39.4±8.7

We were also surprised by the low sol score but it seems consistent with our experience in using it in the field in atopile as agent in our harness. In general OpenAI models didn't do too well on electronics, which seems to change now with GPT-6 Astra. Results are in soon!

We spent a lot time evaluating which model is the best to use in our atopile agent harness and thought the benchmark and results are interesting enough to share.

Evaluating a model for specific use-cases like this that are bit more broad scope than a lot of typical microbenchmarks turned out to be quite the challenge.

Determining deterministically what a "good" electrical design is non-trivial and most electrical-engineers rely on good ol' intuition and decade long experience, so we tried to formalize it.

The only reason it was somewhat possible within a reasonable amount of effort is because atopile provides most of the complicated infrastructure for the benchmark: constraint solver, simulation, code-first electronics modeling, erc & drc checks, high level models of passive electronics components etc

Claude is surprisingly good at discrete digital design with 74xxx, wonder what it trained on. Did you run it on anything else digital?

Love what samuel and his team are building over in the Netherlands! Always thought it would be fun if there was a more beginner friendly version of atopile [1].

[1] https:/atopile.io/

Sourcing parts has been a huge motivation for this. We are still pretty jarred by the supply chain crisis a few years ago. Having the ability to formally verify that an alternative part fulfills the original design constraints is huge.

Not sure whether you played with llm datasheet extraction lately, but they are working their way up there. Even started reasoning about curves and footnotes.

Where do you think is the biggest pain point: part discovery, datasheet verification or substitute determination?

Really excited about constraint solvers making a comeback! Especially after seeing Z3 getting a mention in anthropics fermat post. EEbench is using atopile's internal constraint solver for engineering parameters (think operating temperature, voltage ranges, stacked tolerances etc).

I will never understand why Sonos puts a tweeter in their speakers, but then configure their master chain in a way that it’s basically off. Wonder if they did some focus group testing and people seemed to associate lacking highs with “premium” sound or something.

The next stage in `ato build` process would have caught the missing connections in the DRC check and the LLM could have fixed it itself.

Tried out claude code with atopile this week, and it's absolutely crazy. atopile has mcp support and claude loves using it. The MCP gives access to the atopile design library and compiler amonst other things.

Disclaimer: co-author of atopile here

Finally some love for the best controller ever made. Name of the project is also great. Hope to see some usb version of the dongle hw that supports multiple controllers.