hmaxdml

I thought that fell under BYOD (device) :)

What's reasonable? DBOS has users running queues at millions of tasks per hour.

For fair queuing you can have partitioned queues where only active partitions consume resources

Not saying this in a mean way:

It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.

As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.

The database is exactly the hardcore piece of engineering that's been designed to scale and be fault tolerant for decades

Because you likely already have a database and likely don't need to bring on an entire new distributed system to orchestrate your workflows.

A PG-backed queue is in code right after being in PG, and the beauty of a neat durable queue framework is in exposing it conveniently and efficiently.

That's why their entire business model -- like Astronomer's -- is geared toward cloud hosting. The architecture is so complex it takes a full time team to run it.

Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)

DBOS python supports SQLite. Go is supporting it next release

I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.

Yeah, we've observed that too: people start implementing their own retry logic, idempotency, etc. But then they grow a hard to maintain, complex stack that's not their core business logic. There's a reason why there is a dedicated team building DBOS, every day. Because it's not that easy to build a solid durable workflows engine on Postgres.