(Tedious disclaimer: my opinion, not speaking for my employer, etc)
I'm an SRE at Google, where postmortems are habitual. The thing that jumped out at me here is that a production change was instantaneously pushed globally, instead of being canaried on a fraction of the serving capacity so that problems could be detected. That seems like your big problem here.
(Of course, without knowing how your data storage works, it's difficult to tell how hard it is to fix that.)
I'm an SRE at Google, where postmortems are habitual. The thing that jumped out at me here is that a production change was instantaneously pushed globally, instead of being canaried on a fraction of the serving capacity so that problems could be detected. That seems like your big problem here.
(Of course, without knowing how your data storage works, it's difficult to tell how hard it is to fix that.)