Comment by sgrove

There's a followup study to identify the actual cause of such a surprising outcome https://www.arxiv.org/abs/2506.19823

The combined use of faithful-chain-of-thought + mechanistic interpretation of LLM output to 1.) diagnose 2.) understand the source of, and 3.) steer the behavior is fascinating.

I'm very glad these folks found such a surprising outcome early on, and it lead to a useful real-world LLM debugging exercise!