dpaleka

My understanding is that gwern above linked solid evidence in the paper for more search not being enough, as in, the model's evaluation NN is so way off target when searching, that realistic amounts of search don't help. Go seems to have many possible moves per position, so the search doesn't go very deep anyway.

Feel free to correct me if I'm wrong, it might be that I misremembered how AlphaGo-style systems work.

But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches.

Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)

Why answer questions about harmlessness/safety in such a roundabout way? Both OpenAI and Anthropic are clear about what words like "safe" are intended to mean: a stepping stone to "AI does not kill all people when given control".

Avoiding to state this clearly only invites unnecessary culture war disagreements in every discussion about these models.