pierrekin

I am busy working on an embarrassingly ambitious game reverse engineering project. (To the point where admitting what it is feels like telling someone you’re working on a perpetual motion machine).

It’s a new tool to reverse engineer games in a specific context, and I’ve been using anthropic models to work on it.

I haven’t seen any increase in safeguards triggering or any negative effects in how quickly agents are able to make progress in this space.

To give an idea of scope it’s about 5000 commits, 500k lines of rust code, 100 Ghidra patches of work so far. Made in around 6 weeks worth of 8-15 agents working continuously.

Thanks for posting this, it has made me reflect. I think that’s a fair point.

No, but close. Someone like that built the infrastructure tooling to do deep research, wrote up their process doing that, and then did what you said after, which I consider to be different.

I didn’t read it in full but I spot checked one or two citations and I found them compelling.

I get one a week minimum, hundreds per year. I don’t follow up often enough to know whether they are real and separate etc though.

I’m not sure what you mean? I have goals that I want to achieve; lil ai buddy comes along and helps me, over time buddy becomes better able to help me do stuff.

What do you mean role? Person who does stuff I guess, same as it is now.

Anecdote: As a hapless junior engineer I once did something extremely similar.

I ran a declarative coding tool on a resource that I thought would be a PATCH but ended up being a PUT and it resulted in a very similar outcome to the one in this post.

I would argue that “Why did you do that?” between humans is usually a social thing not a literal request for information.

What the asker wants is evidence that you share their model of what matters, they are looking for reassurance.

I find myself tempted to do the same thing with LLMs in situations like this even though I know logically that it’s pointless, I still feel an urge to try and rebuild trust with a machine.

Aren’t we odd little creatures.

This is something I really hope can be solved.

I long for a “copilot” that can learn from me continuously such that it actually helps if I teach it what I like somehow.

I agree that the model can help troubleshoot and debug itself.

I argue that the model has no access to its thoughts at the time.

Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something.

If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context it recorded (chain of thought for example).

I suspect we are not describing the same thing.

When a human asks another human “why did you do X?”, the other human can of course attempt to recall the literal thoughts they had while they did X (which I would agree with you are quite analogous to the LLMs chain of thought).

But they can do something beyond that, which is to reason about why they may have the beliefs that they had.

“Why did you run that command?”

“Because I thought that the API key did not have access to the production system.”

When a human responds with this they are introspecting their own mind and trying to project into words the difference in understanding they had before and after.

Whereas for an agent it will happily include details that are not literally in its chain of thought as justifications for its decisions.

In this case, I would argue that it’s not actually doing the same thing humans do, it is creating a new plausible reason why an agent might do the thing that it itself did, but it no longer has access to its own internal “thought state” beyond what was recorded in the chain of thought.

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post.

On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that make it harder to see the context and thinking steps, maybe this is an attempt at clawing back that visibility.

Thanks for writing this comment, I think seeing someone’s “first impressions” and then seeing someone else’s response to those thoughts is more interesting and feels more connected socially than just reading a “correct” guide or similar especially when it’s something I’m curious about but wouldn’t necessarily be motivated enough to actually try out myself.

I think it may be that we agree but because the expression “vibe code” has many meanings it’s hard to tell.

I would absolutely still bring a coding agent with me for a project like this, but I would be in the mindset of “I need to understand and be familiar with every line” rather than say, every function signature or every service behaviour.

So it is almost like vibe coding but the abstraction level is lower?

The question I’ve been asking myself recently is, if the act of thinking about the code from scratch is somehow more good than the potential benefit of being able to let that mechanical part be handled by something else, be it another human or an agent.

To be specific I’m referring to a prompt like “next, add a for loop which iterates over the elements in the array and enumerate an index, then call our function $func by reference for each element.” “Is there a more idiomatic way of doing this in $lang?” etc.

This has the advantage to me of letting me code in languages who’s syntax I don’t know or have forgotten, but I’m not sure whether this is trading some sort of short term gain for long term cost yet.