diggs
Joined 14 years ago
Joined 14 years ago
Let me add my own perspective: Distributed locks are a fallacy. They can be beneficial in decreasing contention, under the assumption that "most of the time, only one actor will be active". However, by themselves, they offer no solid guarantees. The blog post addresses this point by introducing the concept of fencing tokens. These tokens have the potential to provide concrete guarantees, but they require the cooperation of downstream systems for enforcement, which isn't always possible.
I was really surprised to see Antirez argue for the probability approach.
In really simple cases you may model this as a one-shot check on a logical collection resource e.g. If I have an organization, and an organization owns many repositories, I may check for action:read on resource:/organizations/:id/repositories. It's very limiting though. What if I want to list all repositories across all organizations I have access to? What if I have multiple levels I want to cut across? Do I need to do all the sub-queries and aggregations myself? Do I need to do a one-shot check on every potential resource? What if the answer is "no" for every one, and I end up doing a table scan of the entire DB just to produce an empty result set? etc.
You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.
Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.
edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?
Without more transparency its easy to imagine something like...
Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in" Step 2. Introduce your own CI solution, which people can now easily switch to Step 3. Lock people in
Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.
I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.
This is very interesting. I naturally use “we” when discussing past products and achievements. I do this in recognition of the fact that things are very rarely designed and built in true isolation. Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward.
“We” is absolutely not a red flag for me.
Ironically I went on to use a lot of ES prinicpals in building a distributed deduplication engine for my previous startup after I left that company. We used a WAL which we were able to rebuild databases/projections from, ship to replica nodes for read replicas, push to S3 for resilient storage etc. it was an extremely powerful architecture for us. It was also complex and required a special skill set to work on and reason about.
I have to imagine anyone that advocates for "moving from CRUD to Event Sourcing" has recently been sucked into drinking the coolaid, and has no idea about the world of hurt they've signed up for, completely necessarily.
In the filesystem example I can visit any node in thee tree and set an ACL on it, optionally recursively. This gives me a lot of flexibility to slice and dice permissions in a complex potentially-multi-tenant resource model. Tags appear to lead to a flattened view of resources - all resources with tag foo are treated the same, regardless of their location in the model. It would be nice to see the author acknowledge this and expand on their examples to see how they might incorporate relational data models.