More search won't do good, but why wouldn't targeted training help? The way I see it is that the adversarial policy search discovers positions which are off-distribution for anything seen in the victim's self-play training.
But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches.
Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)
But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches.
Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)