Keep in mind that it's not just pure learning from zero, with these crazy emergent behaviors coming out. There is a fair amount of "coaching" involved. The OpenAI team explained this last year when they revealed the SF mid 1v1 version of their Dota bot.
People were astounded that the SF mid was creep blocking at an extremely high level, since wave positioning/management is a very complex behavior/process that doesn't pay off directly right away. The OpenAI team said that they basically taught the SF specifically how to creep block, and they've done this with other behaviors as well, like denying allied creeps, as well as a lot of the meta-level strategic play like warding/vision and item builds/usage.
People were astounded that the SF mid was creep blocking at an extremely high level, since wave positioning/management is a very complex behavior/process that doesn't pay off directly right away. The OpenAI team said that they basically taught the SF specifically how to creep block, and they've done this with other behaviors as well, like denying allied creeps, as well as a lot of the meta-level strategic play like warding/vision and item builds/usage.