NovemberWhiskey
Joined 7 years ago
Joined 7 years ago
It may improve efficiency where a protocol doesn’t assure exactly-once delivery of messages, but it cannot help you with problems other than deduplication of identical messages.
Creating a payment is not an idempotent operation. If the economics of the operation can differ when the “idempotency” key remains the same then you’ve just created a foot-gun in your API.
You can document that you’re going to ignore “duplicate” requests that share an idempotency key but that’s just user-hostile. The system as a whole is broken as designed.
Is your point here that every pointer type for which this can be the case should include an explicitly typed null value?
This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).
I don't disagree with you that it's a realtime problem, I do however think that "just" is doing a lot of work there.
I think it often boils down to "know when you're going to start queuing, and how you will design the system to bound those queues". If you're not using that principle at design stage then I think you're already cooked.
In large, distributed systems the best we're looking for is statistically acceptable. You can always tailor a workload that will break a guarantee in the real world.
So you engineer with techniques that reduce the likelihood that workloads you have characterized as realistic can be handled with headroom, and you worry about graceful degradation under oversubscription (i.e. maintaining "good-put"). In my experience, that usually comes down to good load-balancing, auto-scaling and load-shedding.
Virtually all of the truly bad incidents I've seen in large-scale distributed systems are caused by an inability to recover back to steady-state after some kind of unexpected perturbation.
If I had to characterize problem number one, it's bad subscriber-service request patterns that don't provide back pressure appropriately. e.g. subscribers that don't know how to back-off properly and services that don't provide back-pressure. Classical example is a subscriber that retries requests on a static schedule and gives up on requests that have been in-flight "too long", coupled with services that continue to accept requests when oversubscribed.
In NYC, prior to Uber entering the market, taxi medallions changed hands for up to $1mm. Prices were fixed by the TLC.
If these are no strong indications of a cartel, I don’t know what is.