Comment by ag8

Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.