Comment by zem

another, hopefully not accurate, prediction of his was that there is no way to disobey a sufficiently powerful AI in the long run, because it will just factor in the exact differences between what it told you to do and what you actually did, and reverse engineer your behaviour to psychologically manipulate you into doing what it wants.