marcog1

This might not help you, but it might help someone earlier in their career.

I was born, raised and studied in South Africa. Living costs and salaries are cheaper. I worked in California. Instead of living a lavish lifestyle, I saved. In hindsight, I should have saved even more. The important thing I did was opting for cheaper housing. I worked hard, which opened doors. I landed up at a startup that's now doing really well.

I retired after just six years. Six more years later, I'm doing what I want. I've been cycling around Europe and Africa. Next week I'm flying to Canada to explore North America for the next two years. It's a pretty cheap lifestyle, but I get to experience life around the world in a way few people ever do. I'm working on building a presence on YouTube. I've met others who sustain their travels via YouTube. Even if I don't, I can keep going for quite some time living off of savings. I wouldn't be able to do this so easily if it weren't for stock.

I'm not advocating a travel lifestyle. Instead, I'm advocating for saving up while you're earning decent cash. Don't blow it all. Then hopefully you can leave for what you really want to do, and not be tied down due to finances.

You want to replicate as much as possible, but if we ran canary on the same machines we could have testing code bring down production. That's bad.

We actually have a traffic replica (dark client) setup for the new webserver architecture we are gradually migrating to. It likely would have caught this before deploying to users.

We have very comprehensive dashboards. Getting the perfect ones that help in all cases, while not being information overload (the problem here) and being discoverable is a hard, iterative process.

We have talked about it. It is unlikely to helped with an event like this, and I don't recall an event where it would have. It also has the downside of extending our deployment cycle by a lot. Notably, we do run a canary internally, and that had no issues, which actually through us off for a while because while the app was partially down for users it was working for us and that hasn't happened to us in a while.

When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.

We will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.

Every response from our users so far has been thanking is for the transparency. It also represents our internal transparency, and that has a real impact on recruiting.

We do, but it didn't help given the cause of the high cpu was our logging infrastructure (Amazon Kinesis) being overloaded by the webservers.

Correct. We don't roll out during peak load either.

This was the first time we had this class of outage. Many things were in a very bad state, and many of these symptoms were more familiar to us. So we spent time ruling them out before realising webserver CPU was closer to the root cause than the other symptoms.

We roll back by reverting to a previous release on the load balancers, which is usually pretty instant. The previous releases were bad and themselves rolled back, which is a rare situation for us. So there was a bit of scrambling to look into the chat logs to determine a safe (non-rolled back) release we could roll back to. Then the high CPU caused our roll back to be really, really slow. Then we still had old processes running the bad release running, and killing them on webservers with high CPU took a while to actually work. Then it took a bit of time for load to come down on its own. All of this took place within the 8:08-8:29 window reported in the post. And I'm still simplifying a lot.

I work at Asana. There's no way we'll drop our free plan. It's too fundamental to our sales model. We are, however, thinking a lot more actively about what features to make premium. Largely new features we haven't released yet.

Perf is our #1 priority right now. We've had a big team tackle the fundamentals for the last two years, and we're rolling things out in the coming months. I'm personally leading the team that's rebuilding our API on top of this new infrastructure. We're still early in that process, early signs are that we can get perf down to around 100ms. The mobile apps are built on top of the API, so this would make them fast too.