lmeyerov

Those are standalone products and professional services, not internals they use

A problem is some people and teams are world class at a big company... But not everyone, everywhere. For a fun time, read about the rollout and internals of Azure.

Yes, files up the hierarchy to the root, as the app will have broader security etc requirements that all components must abide by (unless they explicitly say otherwise etc).

It's ultimately AI-controlled, so this is more about hints for inclusion, vs being overly prescriptive nor imperative.

We gave a go for cyber on botsbench.com , and this is both more expensive and less capable than Reflection's Beam 1.0 . It'll take a lot of price cuts for us to care on this particular one. Instead, I view more directionally: similar to glm 5.2 -> glm 5.3 flash, I'm curious what Medium 4.1 will look like.

We moved from an ai-maintained wiki in source control (ai_kb/...) to placing specification markdowns right next to the relevant code (src/featXYZ/SECURITY.md, ARCH.md, CONCURRENCY.md, ...). Importantly, we instructed our review/SKILL.md to find and load the spec hierarchy in as context. This helped a lot.

For change justifications, so far we have told the AI to strip change statements from code, and leave that kind of thing to the PR and commit messages. Making that more accessible than hammering git, GitHub can make sense. I'm hoping git protocol and tools catch up to make that more agent friendly, vs the trend of folks making their own new protocols and tools!

Median is < 1d for major ones

We did a small survey of major incidents after shai-hulud and came to 2d cooldowns through that

Awesome

I've suspected for awhile that the frontier models work this way bc from a speed + quality perspective this feels pretty intuitive

yeah for botsbench.com side where we do cybersecurity investigation evals of models + harnesses, we've been having to play with:

1) time budget model gets (5min / task) -- this became a real problem with opus/astra/fable at high+

2) tuning which effort level we report on (when not all)

It's incredible how well talking points on wedge issues get consumed, accepted, and amplified by both bots and people. And even though AI can now detect this in seconds, the problem seems worse nowadays, not better.

(Source: we help power tools used by the big boys to analyze posts like this & their underlying narrative campaigns)

GLM 5.3 Flash even more so... But yes :)

Funny enough, GFQL (oss cypher on CPU/GPU dataframes, no DB needed) runs optimized graph queries with the usual CSR-like indexes... But underlying data is columnar as in duckdb. Likewise, for fast billionscale perf on single node, maps graph compute to pure vector ops. Storage, data representation, & reasoning are largely separated concepts.

If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security

For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)

There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.

I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.