jorgeortiz85
Joined 19 years ago
Joined 19 years ago
The post you are replying to is talking about the small subset of individuals who have had their corpus callosum surgically severed, which makes it much more difficult for the brain to send messages between hemispheres. These patients exhibit “split brain” behavior that is well studied by experiments and can shed light into human consciousness and rationality.
This is one of our few remaining unsharded databases (legacy problems...), so we can't easily canary a fraction of serving capacity. However, one clear remediation we can implement easily is to have our tooling change a replica first, failover to it as primary, and, if problems are detected, quickly fail back to the healthy former primary.
Lesson learned. We'll be doing a review of all of our database tooling to make sure changes are always canaried or easily reversible.
We're always learning and improving. In order to scale, we'll need better ways to manage complexity and isolate failure. Our tools, patterns, and processes have changed quite a bit over the last few years, and they will continue to change. Ultimately, we want every Stripe employee to have the right information evident to them when they make decisions. This will be challenging, especially as we grow, but I'm excited to take on that challenge.
If you're still interested in working at Stripe, I'd encourage you to reapply! Our needs have changed quite a bit since you applied, and we're willing to reconsider candidates after a year has passed. Feel free to shoot me a resume: [email protected]
We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task turns these into alerts to database operators to run pre-defined tools that handle index operations.
In this situation, an application developer didn't add a new index description or remove an index description, but rather modified an existing index description. Our automated tooling erroneously handled this particular change and interpreted it not as a single intention but instead encoded it as two separate operations (an addition and a removal).
Developers describe indices directly in the relevant application/model code to ensure we always have the right indices available -- and in part to help avoid situations like this. In addition, the tooling for adding and removing indexes in production is restricted to a smaller set of people, both for security and to provide an additional layer of review (also to help prevent situations like this). Unfortunately, because of the bug above, the intent was not accurately communicated. The operator saw two operations, not obviously linked to each other, among several other alerts, and, well, the result followed.
There are some pretty obvious areas for tooling and process improvements here. We've been investigating them over the last few days. For non-urgent remediations, we have a custom of waiting at least a week after an incident before conducting a full postmortem and determining remediations. This gives us time to cool down after an incident and think clearly about our remediations for the long-term. We'll be having these in-depth discussions, and making decisions about the future of our tooling and processes, over the next week.
And there are plenty of off-campus jobs (SAT tutoring, etc) that can pay $35+/hr.
The Microsoft world runs .NET, Visual Studio, C#/VB.NET/ASP.NET, targets the Windows desktop runtime (tho increasingly also the web), relies primarily on proprietary (and usually non-gratis) libraries and tools, etc.
The Open Source world revolves around *NIX, uses open-source languages (gcc, clang, Java, V8 Javascript, MRI Ruby, etc), targets the Linux runtime and sometimes OS X/iOS, relies primarily on open-source (and gratis) libraries and tools, etc.
The ecosystem differences go pretty deep. For example, even though either world can interop with practically any SQL database, inhabitants of one will largely choose Microsoft SQL Server while inhabitants of the other will largely choose MySQL/PostgreSQL.
Both can have great software development or terrible software development. It's possible to mix and match (eg, using Windows doesn't preclude you from writing Ruby).
But startups tend to choose the Open Source world, likely due to the combination of lower licensing costs and the "hackability" of open-source software. I'd argue that due to those same reasons, the Open Source world has produced more innovation in the last decade.