Comment by gradus_ad

We have complete access to every "neuron" and "synapse" (crude analogies...) of these models, so in theory we don't need to be so reliant on CoT traces right? I say this not to minimize the difficulty of interpreting raw activations, but I'd expect a huge amount of research to be focused on it. CoT could be obscured by a model outputting language that looks innocuous but encodes actual hidden meaning. Presumably raw activations would be impossible for a malicious model to obscure in this way.