Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor.
The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generator because the main fuel tank pump flooded (AWS was completely unaffected). You just don’t tempt the DR fates like that…
The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generator because the main fuel tank pump flooded (AWS was completely unaffected). You just don’t tempt the DR fates like that…