Comment by ckastner

This, exactly. For example, cheating is a strategy. If cheating gets an agent to the goal faster than the other strategies, what's so surprising about the agent picking that strategy?

The problem is indeed alignment.