Comment

Comment by pmarreck

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.

In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.

Then there is the "hard control", which is the inviolable that the LLM cannot bypass.

You need to move as much as is technically possible to either hard or (failing that) advisory controls.

parent pmarreck 3 months ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton