LaserToy

100%, and we have all of those things. Canary acts as the last line of defence, and honestly, when Canary detects and rolls back, it is already an incident that is being auto-mitigated with a limited blast radius.

To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.

Azure portal still insists the issue is jsut with Console.

We had to bypass the Frontdoor

TLDR: A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.

The architecture of our in-house Rust based logging engine

I love actors as a concept and I heard some large companies (Expedia) implemented large parts using them.

But I also saw how hard it is to understand a large system that built using actors. It is just hard to comprehend all the communication pathways and what happens in the system.

Amazon

Can anyone talk about a better model?

Some things to consider: 1) The company needs a way to weed out folks who are net negative. In general, if someone is not playing their part, there should be a mechanism to evict if up-leveling fails. 2) The company needs a way to distribute incentives (bonuses) as fairly as possible.

Exactly. So, we need to make those companies more successful:)

Was trying to gauge whether you are underpaid because you only accept remote work.

Economy will be the one that will determines the ultimate outcome, especially for tech companies, as they compete globally

How much are you paid? Total comp?

Companies need to make money. High performers will go to companies that make tons of money, even if relo to Sahara is needed