cjbarber
Joined 13 years ago
Joined 13 years ago
This is based on the career philosophy around joining breakout startups.
If you have suggestions of how I could improve usability or suggestions for the future, let me know.
> We evaluate 11 frontier models (see Figure 1), all run within the OpenCode agentic harness. Claude Opus 5 performs the best, yet still only manages to pass 12.3% of tasks. Claude Fable 5.1 and GPT-6 Astra are close behind, but the other eight models have significantly lower pass rates.
> Our general approach is to automate the experimental loop. We think this approach is broadly applicable across many different fields of science and engineering. We’ll initially focus on ML research and engineering, but believe the approach can help with important subproblems in nearly every one of the fourteen <at>NAE Grand Challenge problems. We think doing this well requires strong expertise in machine learning as well as large-scale systems.
See also: https://www.nae.edu/20782/grand-challenges-project
Those 14 are:
NAE Grand Challenges for Engineering
1. Make Solar Energy Economical
2. Provide Energy from Fusion
3. Develop Carbon Sequestration Methods
4. Manage the Nitrogen Cycle
5. Provide Access to Clean Water
6. Restore and Improve Urban Infrastructure
7. Advance Health Informatics
8. Engineer Better Medicines
9. Reverse Engineer the Brain
10. Prevent Nuclear Terror
11. Secure Cyberspace
12. Enhance Virtual Reality
13. Advance Personalized Learning
14. Engineer the Tools of Scientific Discovery
I think it'd be good if there were more products that did a better job of making an actually undetectable agent, but doesn't seem like any exist yet.
[1]: https://usefoil.com/research/stealth-browser-leaderboard
Other products in the space:
- Foil (https://usefoil.com/), I'm biased, a friend is building this
- Kasada https://www.kasada.io/
- DataDome (https://datadome.co/)
- Castle (https://castle.io/)
- Fingerprint (https://fingerprint.com/)
- HUMAN (http://humansecurity.com/)
- Google Cloud Fraud Defense, which is basically the updated reCaptcha (https://cloud.google.com/security/products/fraud-defense?hl=...)
- this, Cloudflare Precursor
It seems like some of the main reasons people care so far are:
- Preventing automated credential stuffing
- Preventing bots from creating a bunch of fake accounts (eg free trial abuse, which can also lead to high twilio SMS bills!)
- Reducing payment fraud
- Blocking LLM scraping
- Blocking automated scalpers (!) eg for tickers or sneakers
I'm curious to see which use cases end up dominating as the reason companies care about this. And I'm hopeful that my agents will still have good ways for me to browse and do things on the web on my behalf - eg detect agents and route them to an agent path, rather than blocking them.
(I'm interested in tools for detecting AI agents and seeing how this shifts as bot traffic goes way up.)
https://x.com/chrisbarber