gwen-shapira
Joined 6 years ago
Joined 6 years ago
I explored a bit what dropping columns actually does (mark the column as dropped in the catalog), what VACUUM FULL cleans up, and why we are still (probably) compliant with the GDPR.
At this point, all transactions go through a single transaction coordinator that uses 2PC to coordinate all transactions. This makes many aspects of coordination fast, reliable and simple. It works well because OLTP systems typically have few DDLs an hour, if that (although we tested with hundreds of DDLs per second, just to be sure).
When we look at other types of transactions, OLTP systems need to handle many orders of magnitude more. Even small apps typically have hundreds of transactions an hour, while large systems require millions of TPS.
Getting from the current architecture to something that handles all transactional load of an OLTP system will be a journey. And arguably one better served with a different architecture, one not based on 2PC.
But AI companies, that tend to scale faster, also seem to benefit the most. So we wanted to make this clear.
You are right, it is hard to cover everything in one blog...
But the gist is that schemas and all the objects in them have few significant drawbacks:
1. Catalog overhead. Each schema has all the objects. PG Catalog doesn't scale to huge numbers. If you test anything that needs to work with the entire catalog, you'll run into issues in the 4-5 digits range (depends on number of tables, indexes, etc).
2. If you need to write a query that executes on all schema and aggregates results (for a report or something), you need to build your own tools.
3. Schema migrations and DDLs are increasingly hard to coordinate (I gave a whole talk about that here: https://www.youtube.com/watch?v=Nnz4VesXyUA
We basically didn't just build the distributed query and DDL layer, we also did some smart things in Postgres itself (blog TBD) that allow us to reduce the catalog overhead significantly.
You may be under-selling LittleHorse a bit (here and even in the README). Few things I find cool about it:
- How easy it makes writing human-in-loop workflows (or these days, agent-in-loop). Everyone wants human-in-loop, but very few workflow frameworks make it simple.
- Using LittleHorse to introduce stateful guardrails to agentic flows. This helps with context compaction and other safety issues.
- The new (experimental?) generation of workflows from JSON. This simplifies things quite a bit.