coops

Someone like Amit doesn't need money; by the time you get as highly placed at Google as he was you will be wealthy. I believe that Uber has interesting engineering challenges but likewise have doubts about their profitability.

It's a real piece of hardware.

- crawlers

- websearch

- google brain

- video transcoding for youtube

- appengine (think snapchat)

- gmail

Almost everything Google does runs on Borg.

The best you can do is two-phase commit.

SRE here. If you can handle your work nobody cares how much you are in the office. At all. However, unless you are insanely organized you won't be able to do this every single week--most successful googlers have several projects going on at once and there will be some times when you need to work hard on more than one.

You cannot hire top-quality engineers who are willing to do shift work.

Yes, this is certainly a state you would desire to avoid. However, you didn't answer any of my questions.

Can you go into more detail regarding the prohibitive consistency issues? How do you maintain consistency in steady-state (ie. not during migrations?) Also, how do you make the call as to whether to bring your site down vs. attempting a live migration?

It would be interesting if someone from Github could discuss why they chose to do this migration by taking the whole site offline and doing the migration all at once. Did anyone investigate if this could be done without taking the site offline?

What's it like to administer a cluster of Mac Minis? Do any of the popular open-source orchestration tools work?

Consider that if you collect device fingerprints, you can detect users who live together, because they are likely to share devices. If somebody bad then gets access to that data, they could do creepy things to your users.

I think what will actually happen to requests in flight is:

- partial data received by old HAProxy is lost as old HAProxy exits

- new HAProxy comes online, binds to port, receives fd

- iptables rule removed. new HAProxy starts receiving new requests

- in-flight requests from the old HAProxy are timed out by the kernel (TCP RST) as nothing is there to read request data from the old fd or send response data.

So I think this is actually "worse" in some sense than the other retry behavior since it's not recovered inside the same TCP session but instead forces the client to open a new TCP session.

I feel I must point out that this relies on your clients to retry packets that iptables drops, so at the least they'll have a slower experience than they otherwise would. I believe that this will also break requests that are in flight when the reload happens, if they have sent partial data.

Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.