Our incident management includes chatops automation that, among other things
- generates a per-incident Google Doc
- opens a per-incident Google Hangout
- pages appropriate people via PagerDuty
During this incident (which affected our customers who use Google to log in), people had difficulty accessing all three (our PagerDuty uses Google Oauth).
This was a good reminder to have alternatives for all crucial incident management services. (We have backup plans in case Slack or our automation scripts are down, but didn’t have backups for Google...)
- generates a per-incident Google Doc
- opens a per-incident Google Hangout
- pages appropriate people via PagerDuty
During this incident (which affected our customers who use Google to log in), people had difficulty accessing all three (our PagerDuty uses Google Oauth).
This was a good reminder to have alternatives for all crucial incident management services. (We have backup plans in case Slack or our automation scripts are down, but didn’t have backups for Google...)