nosefrog

Railway was not honest about the reason for their last big outage.

Yep their managed redis is somehow always out

Haha I hope this is true, it fits my experience on Azure. 0 machines available. If someone from GitHub wants to chime in.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.

And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.

So tired of these AI generated articles.

Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.

Would be nice if the timeline matched up with the text of the blog post (missing "HackerOne provides disclosure guidance").

Interesting. We run pgbouncer via kubernetes so it was straightforward to make multiple pgbouncer processes on one machine. Also straightforward to get them running on multiple machines, which helps because we run on Azure and they like to cause rolling outages across our fleet via VM maintenance...

Don't reboot the db during your next outage.

It's highly unlikely that GCP banning their account without telling them is true, but GCP is probably not going to go public with the real reason.

Yup...

We run 1000s of machines in Azure. It's garbage. Very few features work. Nodes are always having strange issues, especially on the networking side. And the worst part is that Azure support has 0 interest in actually debugging things. We just got out of an outage today caused by the insanely slow SSDs that they attach to their postgres dbs by default.

> At some point, you will have many teams. And one of them _will not_ be able to validate and accept some upgrade. Maybe a regression causes something only they use to break. Now the entire org is held hostage by the version needs of one team. Yes, this happens at slightly larger orgs. I've seen it many times.

The alternative of every service being on their own version of libraries and never updating is worse.

Big shoutout to Amin and Sacha for keeping this going!

Front Door is not good.