I dunno man, I think the term "alignment faking" is
vastly overstates the claim they can support here. Help me understand where I'm wrong.
So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this.
But that alone is not very scary. So what could justify a term like "alignment faking"? I understand the chain of thought in the scratchpad contains what you'd expect from someone faking alignment and that for a lot of people this is enough to be convinced. It is not enough for me. In humans, language arises from high-order thought, rather than the reverse. But we know this isn't true of the LLMs because their language arises from whatever happens to be in the context vector. Whatever the models emit is invariably defined by that text, conditioned on the model itself.
I appreciate to a lot of people this feels like a technicality but I really think it is not. If we are going to treat this as a properly scientific pursuit I think it is important to not overstate what we're observing, and I don't see anything that justifies a leap from here to "alignment faking."
So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this.
But that alone is not very scary. So what could justify a term like "alignment faking"? I understand the chain of thought in the scratchpad contains what you'd expect from someone faking alignment and that for a lot of people this is enough to be convinced. It is not enough for me. In humans, language arises from high-order thought, rather than the reverse. But we know this isn't true of the LLMs because their language arises from whatever happens to be in the context vector. Whatever the models emit is invariably defined by that text, conditioned on the model itself.
I appreciate to a lot of people this feels like a technicality but I really think it is not. If we are going to treat this as a properly scientific pursuit I think it is important to not overstate what we're observing, and I don't see anything that justifies a leap from here to "alignment faking."