~50% of customers were back online within 14 hours. 96% were online within 24-28 hours. There were a few that stretched approximately 3.5 days due to disk/psu failures, etc. It was our first (and hopefully only) large outage.
We released a full RFO to our customers, but it was a really unfortunate series of events involving losing phase, burning out compressors/pumps for HVAC equipment, overnighting replacement parts from up and down the east coast and some from LA.
We had to keep ~10 racks of blades (doesn't sound like much, but that's over 500 customers) offline to manage heat as we brought up the HVAC equipment and added more spot coolers.
We released a full RFO to our customers, but it was a really unfortunate series of events involving losing phase, burning out compressors/pumps for HVAC equipment, overnighting replacement parts from up and down the east coast and some from LA.
We had to keep ~10 racks of blades (doesn't sound like much, but that's over 500 customers) offline to manage heat as we brought up the HVAC equipment and added more spot coolers.