> Who is surprised here it took them this long to recover?
They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down…
So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.
They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down…
So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.