darkamaul
Joined 5 years ago
Joined 5 years ago
- Aug 6: Off By 1, the security lab of 1Password publishes their blog post (current title: Why AI-generated vulnerability patches still require expert human review)
This article gets some specialized and mainstream press coverage
- September 15 : Trail of Bits publishes a rebuttal on their blog 1Password's AI patching benchmark is misleading
- September 22: Suha publishes the current blog post
The main idea in the rebuttal is that the headline number (26 percent only of the AI generated patches are flawless) is misleading because the experiments were not well designed.
Instead of forwarding ANTHROPIC_API_KEY / OPENAI_API_KEY into the guest, we simply run a reverse-proxy within the VM that replace a constant-time token with the real API key on the host.
Works well enough in practice
I would think this is one of the most critical dependency that you would not want to get right away at let it rest for a couple of days
I would recommend anyone that has desire to provide stricter boundaries to agents to give it a try - while remembering that a motivated agent would probably be able to escape it :)
And as always with those satirical posts, I love the attention to the details (location: Nebraska for instance)
Why spend time and resources creating a complex orchestration workflow around a model limitation, where the next model might just behave better without all these constraints around it?
Anyone can point an LLM to a code base and ask to find a vulnerability - and the initial set of findings is going to be rather lame.
Encouraging researchers to stick to a target and to report 7 lows before getting in will probably make their contributions more valuable.
Sure they are sometimes some buggy UI controls (like the focus not working), but for the most part, the experience has been smooth.
Did you try to report the bugs when you had some? Maybe this could help everyone else using the browser.
While the EU loves its regulation, I still feel it’s too early to write it down in the AI race. It will not replace Anthropic or OpenAI any time soon, but even Google and Meta fail to do that.
If AI continue to grow and expand, there is enough space for many more unicorns.
[0] https://techcrunch.com/2026/05/28/why-paris-may-be-the-most-...
At the end, I admire the craft and patience to try to solve code diff rendering, and wish the folks at GitHub could put the same effort to improve their platform.
On a side note, I feel that we’re going to see more and more of this type of agentic usage, in well defined sub tasks, and the ability of a model to try many possibilities is a huge gift here.
No memory corruptions, no heap shenanigans, just pure logic bugs, almost made easy to understand for the reader.
As a bonus, they also posted a reverse chain: popping Edge from calc [0].
[0] https://www.youtube.com/watch?v=Bh1zglq4mLE