afc

+1. This is a very real problem in practice.

A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).

We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).

Haha, tell me about it! I spent more than a decade mostly just deploying these systems across just one company. It's far from trivial!

Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.

Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.

Interesting perspective. Your first paragraph also describes my experience, perfectly.

However, even though I also think nothing comes even close to parenthood for me, my perspective is that society strongly encourages —I'd even say strongly pressures— people to have kids, to feel that if you don't, it's almost like something is wrong with you. Maybe we're just different generations? :-)

Thank you for taking the time to type that, I found it informative and interesting. Like we like to think of it as a single idea that worked well, but it's actually a ton of work to get a million details just right, and failure in just a few of them would just have made the whole endeavor (of the printing press) fail.