WookieRushing

These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies.

I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc

Yup! This is the reason why its so cheap for them. Other companies in similar positions have cache nodes in the ISPs and this dramatically lowers the cost

Why do you think its easy?

There's a lot of systems where you can easily take down some hosts, but taking down more than N% at a time causes issues. If your fleet is large enough then you are limited by the largest set of hosts where you can only take N% down at a time. Now you could say keep the sets of hosts small or N% large. But that can cause other issues as you typically lose efficiency or zonal outage protection.

A solution to this could be VM live migration or something similar. This breaks down for storage systems where you can't just migrate those disks virtually since they're physical disks or places that don't use VMs.

My bet is on cache servers having a bad release

It’s millions a year and ms teams is near free. In a cost cutting environment this is near perfect.

Why is there input lag?

I’ve used something identical and mosh makes this just work. Most devs at that company swear by remote builds and hate laptop builds

They can get it from your android phone or from chrome if you’re running it

Its like Fish shell vs Bash shell.

Bash has weird defaults so you end up googling for everything. In fish, it just works and you barely need to search for anything.

Sane defaults matter. With hg, I don't need to struggle to get it to do what I want, it just gets out of the way. With git, sure it works but like you said it has a bunch of ducktaped tools together that change the defaults or just generally make things easier.

Now hg is half the pattern here. The other half is stacked commits. Each commit should build and get reviewed separately. There isn't any waiting for reviews on each commit, they all get reviewed over time and you rebase any changes that are requested. With git this is amazingly painful and half my zshrc is about making this simple. With hg, it just works. Take a look at hg absorb or hg split, theyre features built on top that yeah can replicated in zsh scripts but its kind of nice when you can assume they just work. It means junior engineers don't spend hours trying to fight git with stacked diffs.

Sapling is trying to fight the network effect here by doing the classic built a compatible but legitly better front end. Compatible with github but sane defaults is a BIG thing.

When risk to society becomes high enough then it becomes necessary.

See paying taxes, conscription, jury duty

Though judging the level of risk is very hard and there is often not a clear line of when force is needed.

What psaux mentioned makes more sense. A node == one Cassandra agent instead of a server.

Past 100k servers you start needing really intense automation just to keep the fleet up with enough spares.

If you’ve got say 10k servers it’s much more manageable

The fun thing is Cassandra was born at FB but they don’t run any Cassandra clusters there anymore. You can use lots of cheap boxes but at some point the failure rate of using soo many boxes ends up killing the savings and the teams.

I was also surprised by this. 300K nodes for a distributed DB is kind of crazy. I’ve worked with similar systems but they stored much more than 100 PB with 10x less nodes

Apple is using less than one TB per server…

But when you see the 1000s of clusters it starts to make sense. They probably have a Cassandra cluster as their default storage for any use case and each one probably requires at least 3 nodes. They’re keeping the blast radius small of any issue while being super redundant. It probably grew organically instead of any central capacity management