Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.
Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.