brunocalza

This is interesting. Thanks for sharing more. Looks like it's a trade-off of it being relentless, which is something we want in some cases. We need to figure out a way of closing the door and at the same time signaling the door is closed so it does not keep trying different manners.

I've been thinking about this but in a different context: non-coding agents. e.g., an AI agent that approves travel expenses is not allowed to approve expenses bigger than X USD (no matter what). In this case, it looks closer to an ACL thing.

I think we should aim to move these security violation rules away from the prompt, to a deterministic place. Not sure how your orchestrator works, but is it possible to add a check between the agent's decision and its execution? e.g. the agent decides to read a file, that decision goes somewhere that checks if the agent has permission to do that or not before it actually reads the file.

Every relation will probably go down that path. Building trust will be interesting.

Yeah, totally

> We can't see the code for the laws of physics, and yet experiment by experiment we've come a long way.

Interesting perspective. Haven't really thought through that lens

> Then I compared the implementation, for correctness, to other systems, finding that other implementations sometimes contained more errors. I researched more, and found that the local inference world is full of subtle errors that accumulate and damage the model output, issues in the attention implementation causing performance slopes after the context is over a certain limit because indexed attention implementations are broken (do more work than they should, for instance), and so forth.

I agree 100% that AI helps a lot with that. But I feel like there's something missing between "AI helps a lot with that" and "I believe reading code is mostly pointless". I genuinely wonder how the above can be accomplished without reading any code.

The idea that you need to motivate yourself to do a thing you don't want to do is an idea that needs deeper investigation. I've caught myself trying to do that a bunch of times. Why the hell I think I need to do this thing in the first place?

I totally get things like I have a job and there's a task that needs to get done. But what about outside the job life?