brene

Eleanor from my team and I collab'ed on this benchmark. A few things worth stating up front because they shape how to read this. We constantly see LLMs saying "THIS IS A CRITICAL SECURITY FINDING" and we wanted to put it to the test. Most LLMs naturally gravitate towards using CVSS for severity scoring.

We wanted a task where model judgment could be checked against verified results from security engineers (ground truth) and root cause "why" security findings are always inflated. TL;DR LLMs still make too many severity judgements without the proper context, so naturally they bias towards the "worst case" scenario. Happy to answer any questions on this topic

How does this compare vs. Turbopuffer?

Rene from Casco here. While our agents were performing a security test, they discovered a database takeover vulnerability. It's a good example of how SQL injection is still a test path that needs to be explicitly be validated. Really want to give props to the ElectricSQL team from issue reported to issue fixed and deployed, it took ~2 hours.

Just debugging the issue :-)

are you using Safari's Lockdown Mode?

Author here. We were analyzing a compromised contributor account targeting better-auth when we noticed something interesting about the attack vector. Most coverage of supply chain attacks focuses on the "what happened" but I wanted to document the "how it actually works" with the deobfuscated code.

Wwo things stood out: 1. hiding the payload in next.config.mjs is clever because GitHub's UI truncates long lines so the malicious string is literally invisible when scrolling through the file. second, storing the c2 payload on binance smart chain means theres no server to take down. The axios attack was mitigated by removing the GitHub-hosted payload. This one can't be.

2. found 30+ repos with the same signature string. Pretty sure there's way more we didn't catch with basic string matching.

happy to answer questions about the deobfuscation process or the c2 protocol analysis.

Do you see this project merge with the Chonkie at some point? Or do you intend to keep it separate?

How does it deal with loops? I’ve often see workflow builders struggle at that?

I actually wonder is there a way to feed back some consistently reedited code into the context window of your coding agent tools, so that future edits require less tokens?

Hi Rene from Casco here. I think the post just referenced us as a customer because we use it for pentesting. For us, Prism solves the "browser agents can reliably auth into any website" problem.

Hi - Rene from Casco here. Thought to share a bit about our journey of dealing with auth for browser agents before Prism. We have a diverse set of customers whose login experience differ dramatically. Sometimes it's directly accessible on request, other times, you have to click through into a "login menu", other times we'd be dealing with Google sign-in and OTP.

We initially tried manually uploading session cookies to our browser agent after we authenticate locally. But soon realized how unscalable that is. We needed a general purpose API that allows our agents to auth into any application reliably. We needed something like Prism because making an agent reliable for our vertical is hard enough and I don't want us to maintain infrastructure just for the purposes of managing test user credentials and session management. If you're using browser agents and they've "hit the auth wall", then you know what I'm talking about.

Thanks for building Prism for us and letting us be a pilot customer. The API is straightforward and a pleasure to use. Can't wait for user sign-up and GitHub auth support to come soon.