ssalazars

I see what you did there :)

Correct me if I'm wrong, but you would still need to pay $3 USD/mo for Fastmail even if you use 1P. Whereas with Relay, it's 0.99 USD/mo, and no need to migrate my existing email to any other service.

The problem is, I don't have access to the email used to create this account. I created this account long time back, when hotmail was a thing. I haven't logged into that email account since 2004. I tried getting access to it, but wasn't able to do so. So basically, this account is in a weird state where it exists, but I don't have access to it.

I'm also working on a Java + Dropwizard project. MongoDB is the option I went with as a DB, and SendGrid for transactional emails.

Did you evaluate Digital Ocean as an option?

I understand it. I've worked in AWS, and now in OCI, dealing with systems that affect hundreds-to-thousands of customers, which businesses are at stake.

Mitigation is your top-priority. Bringing the system back to a good shape.

If there needs to be follow-up actions, take the less-impactful steps to prevent another wave.

If there was a deployment, roll-back.

My concern here is, a deployment have been made months ago, and many other changes that could make things worse were introduced. This is the case. The difference between taking an extra 10-20 minutes to make sure everything is fine, versus taking a hot call and causing another outage makes a big difference.

I'm just asking questions based on the documentation provided; I do not have more insights.

I am happy Stripe is being open about the issue, that way many the industry learns and matures regarding software-caused outages. Cloudflare's outage documentation is really good as well.

Thank you for taking the time to respond to my questions. I believe the high potential of causing a follow-up incident was left out of the post (or maybe I missed it?).

I hope that lessons are learned from this operational event, and invest towards building metrics and tooling that allows you to, first of all, prevent issues, and second, shorten the outage/mitigation times in the future.

I'm happy you guys are being open about the issue, and taking feedback from people outside your company. I definitely applaud this.

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout.

There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was back to normal? How can they decide, and document the change within 20 minutes? Are they using CMs to document changes in production? Were there enough engineers involved in the decision? Clearly all variables were not considered.

To me, this demonstrates poor Operational Excellence values. Your first goal is to mitigate the problem. Then, you need to analyze, understand, and document the root cause. Rolling-back was a poor decision, imo.

.../.zoomus/ZoomOpener.app/Contents/MacOS/ZoomOpener

HOLLY SH*T! This is insane.

I currently work for Oracle Cloud Infrastructure; ex AWS for Commerce Platform and Identity organizations. Let me tell you, OCI has a group of brilliant, industry mature developers and people. I've been working here for 2 years, and I've never been happier before. This org is nothing like Oracle Corp; started by ex-AWS/MSFT people, the environment feels just like those companies + a well funded start-up hype.

And the best thing, no assholes and backstabbing like in AWS. :)

Will there be documents in Spanish or other languages soon?

As a mexican living in the US couldn't agree more.

Need an ultra sound? Wait for 3-4 weeks. Want an MRI? Wait for 2-3 months.

Why can't I get immediate care when I need it without having to step on the emergency room and pay (probably) thousand(s) USD?

I've thought about this problem, specially in LATAM countries were gov agencies are still ages behind in terms of security/compliance and sharing these type of documents. There is definitely a need for this.