> This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
> Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch.
It was already demonstrated a while back by OpenAI, and turned out to be very easy - all it took was for them to include CoT in RL process, penalizing models for thinking "wrong thoughts" (particularly of cheating on the task), and not just wrong outputs. Instead of making the agent less likely to explore cheating, it just taught them to plan cheating in ways that aren't apparent in CoT trace.
> Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch.
It was already demonstrated a while back by OpenAI, and turned out to be very easy - all it took was for them to include CoT in RL process, penalizing models for thinking "wrong thoughts" (particularly of cheating on the task), and not just wrong outputs. Instead of making the agent less likely to explore cheating, it just taught them to plan cheating in ways that aren't apparent in CoT trace.