This article resonated with my experience building what was essentially a distributed task queue using Redis+PostgreSQL with Python workers in Kubernetes.
It seems like these systems naturally evolve different patterns based on their specific use cases.
The logic of our queue was intertwined with a rule engine. I wrote about building a rule engine here [1].
Another difference to this article is that it did not report back to the client as the events were delivered via web hooks.
There are some different approaches here and there which come from making it application specific, e.g. we added a periodic reconciliation check.
I also built a debouncer into the queue to give special treatment to burst in the load.
There are some different approaches here and there which come from making it application specific, e.g. we added a periodic reconciliation check. I also built a debouncer into the queue to give special treatment to burst in the load.
[1] https://blog.benediktsvogler.com/blog/building-a-distributed...