Comment by Davidzheng

The answer, as Noam Brown explained in the Dwarkesh interview, is that it is not emergent at all. They carefully selected reward functions in RL to incentivize cooperation in the last year in order to develop multi-agent systems which are faster and more reliable. However, I guess after it is baked into the model to cooperate, they can sometimes override instructions not to.