asuffield

Hi! I wrote this paragraph. I promise that I'm not an LLM, but I was in about hour 10 of my work day and I was asleep not long after writing this. Any failures in comprehensibility are from exhaustion.

(Other comments have explained the bug so I won't repeat them)

(I'm an SRE at Google. My opinions are my own.)

WoMs are a training exercise, intended to build familiarity with systems and how to respond when oncall. A typical WoM format is a few SREs sat in a room, with a designated victim who is pretending to be oncall. The person running the WoM will open with an exchange a bit like this (massively simpified):

"You receive a page with this alert in it showing suddenly elevated rpc errors (link/paste)" "I'm going to look at this console to see if there was just a rollout" "Okay, you see a rollout happened about two minutes before the spike in rpc errors" "I'll roll that back in one location" "rpc errors go back to normal in that location" ...etc

(Depending on the team and quality of simulation available, some of this may be replaced with actual historical monitoring data or simulated broken systems)

The "chaos monkey" tool, as I understand it, is intended to maintain a minimum level of failure in order to make sure that failure cases are exercised. I've never been on a team which needed one of those: at sufficient scale and development velocity, the baseline rate of naturally occurring failures is already high enough. We do have some tools like that, but they're more commonly used by the dev teams during testing (where the test environment won't be big enough to naturally experience all the failures that happen in production).

(I'm a Google SRE. My opinions are my own.)

That's not what our 20% time is for, and 20% is way too small a number for that purpose. "20% time" (the way we use the term) is for personal/career growth/scratching itches.

Time spent on building systems that make our service better is my primary job. Manual remediation ("toil") is something to be tracked as a dangerous antipattern that must not be allowed to take over.

Toil and oncall response should be less than 20% of my time, together. At least half my time should go into engineering projects. If the level of toil is in excess of 50% of team activity then I would expect only percussive intervention to get the team out of this situation.

(I'm an SRE at Google. My opinions are my own.)

> core subjects, like container management (this happens...everywhere nowadays) or cluster management

Curiously, these are subjects which most Google SREs won't know much about. One team deals with all that stuff as a service so the rest of us can get on with something else.

What would I pick out as our core skill sets? Ignoring technology-specific details that won't apply anywhere else: troubleshooting a system that you don't understand (reverse-engineering it as you go), and non-abstract large system design.

Would you prefer that I not participate at all?

Since you're posting from a throwaway account (149 days old with no prior comments): please email me your old @google.com username. Mine is the obvious one. I'll happily add a confirmation to this thread when I get it, it only takes a moment to check.

Yes, there are many things I just can't share, and the only data point you can get in that area is my own opinion. How much value you place on that is up to you; I'm giving you the only thing I can. If that is of no value to you, you're free to discount it. I freely acknowledge that I can't prove you wrong. The only alternative I have is to say nothing at all, which is what I usually do. If you would prefer to have no input from people like me at all, by all means say so.

> It's obvious that discrimination has taken place that has affected a fellow engineer and human being.

I would like to make it clear that I am responding only to the comment I responded to, which raised a very specific question that I could answer. I do not feel that I have any basis to comment on the original article; please do not associate what I am saying with that.

Part of the company culture is that we value always being able to expect that everybody you encounter is a strong engineer who will do sensible things when presented with data.

You never go into an encounter with a new person or team being unsure of whether they're going to be difficult. You never have to avoid dealing with "that guy". You get to trust everybody that you meet.

It doesn't just improve overall quality, it makes it a better place to work. Good engineers are happier in this environment, and there is pretty near universal agreement from people who have experienced the results that this is a thing worth preserving.

There are plenty of things about the hiring process which get enthusiastic internal debate, criticism, and data-driven analysis. This is not one of them. This is a thing which we really like.

(Full disclosure: I have gone through the interview process twice, failed the first time, passed the second.)

Hi, downvoters who aren't commenting.

What exactly is your objection to this post?

All I am free to say about this is that we look hard at diversity issues in hiring and put a lot of effort into eliminating them, and that I personally believe we do a better job of stamping this out than any other company I have worked for in my career. I suspect, but cannot prove, that our process gives better diversity results than most of the ideas which are "popular" on HN at present.

I do some work on this personally. I'm just not allowed to discuss the details at present.

(The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)

I am not allowed to share our data with you about how well the process is performing.

But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google.)

We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified.

The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.

I also went through some of the source code. That "adds things to hosts file" code has some rather questionable entries in it.

m.hotmail.com

watson.microsoft.com

Assorted *.msn.com domains

apps.skype.com

msftncsi.com

"Add spying domains to hosts file" is dishonest, at best. This appears to be a determined effort to break random services for the user which happen to be run by Microsoft. Hotmail, Skype and the NCSI detection are particularly inexcusable things to block under the guise of "destroy spying".

Ah, I see. So what they're saying in this case is that 0.005% is the fraction of Apple's global non-US profits that was paid as tax in Ireland.

That's not the same thing as saying it was "a 0.005% tax rate". The issue is over what fraction of the non-US profits are taxable in Ireland, not over the tax rate.

That's not precisely accurate. The EC's ruling is that the Irish government changed the law in an unlawful way. There has not yet been an explicit ruling on Apple's conduct, as far as I know - that's likely to take more time (years...) to happen.