dhorthy

thanks. We have done a ton of red team pentesting with agents, both internally and through trusted partners. We're doing more continuously as we go.

yes, crazy to think that the original RPI hype was just over a year ago.

or someone could clone the SCB repo and make a fork that proves it one way or another

yep agree

they yearn to create content, generate more tokens, etc etc - agree

Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range

The part II post linked in my top comment has a list of the challenges I used for that.

To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running

great post - the same thing that has always been true of software engineering is still true:

Do fewer things in parallel, do them very well, optimize for getting incremental value out to users, FOCUS

It's been 30 years since the XP and Lean folks said "limit work in progress (50+ if you count the eli goldratt thing) - its still true

> AI-written emails where the commenter would rather see the prompt

this a thousand times this

LLMs are information transformers. if you're trying to take some rough idea and blow it out into something that "feels" substantive, you're just combining your high-signal idea / prompt with a bunch of meaningless bloat and noise from the model weights.

they seem to be able to do the call-stack diff stuff pretty well still