nikita

Today CDC, tomorrow an authoritative part of storage.

If you product is CDC based (peerdb) you don’t want storage to support this :)

This architecture is better for OLTP because all maintenance operations are moved to storage AND it has all other benefits such as LTAP that emerge from having a scalable storage.

Conversion is async. The whole point is to never deal with CDC which is error prone and taxing Postgres with occupying a replication slot and burning memory and cpu in the OLTP system.

Recent data plus working set is always in Postgres page format.

Historical data when pushed to s3 is in parquet. This happens async - not on the transaction hot path.

So older data below certain LSN is on s3 in parquet available to all analytics processing. Hot data is on page servers in page format for OLTP.

You can be smart in querying both representations for real time analytical queries

Safekeepers keep a window of WAL in Postgres WAL format and doesn’t have an external API.

It streams WAL to pageservers and Postgres read replicas

* Rate limiting on proxy in front of compute fleet

* Large tenants are broken up into shards, reducing hotspots

* Each shard is throttled to a fixed req/s rate

* We do not run pageservers at their redline in terms of CPU load, so there is some slack to take up bursts

* Capacity quotas which selectively throttle write traffic to the largest databases if they are competing with others for disk space, until the larger database is migrated away.

Generally the more throughput the system supports the better. In this case we were hitting limits (btw each operation is many queries of different sizes) and the customer observed higher latencies which is typical if the system can't sustain the throughput required.

After this change latencies are back to normal and throughput increased.

Lakebase is referring to the fact that in addition to disaggregated storage s3 is authoritative storage for older data.

Since data is on s3 (or lake) you can perform direct to s3 type operations like data loading, reading this data by engines that are not Postgres and more

Lakebase is OLTP.

We provide you fully managed Postgres. Lots of our customers use it for lots of small instances of Postgres since using Lakebase is so lightweight.

Small and large instances benefit from this performance optimization.

This applies to our storage implementation. In Lakebase architecture storage serves pages and it doesn't always have the most recent version of the page and therefore it reconstructs it on demand.

In the past we relied on Postgres compute to periodically send a full page so reconstructive a page was always a bounded process. Once we turned it off (and got all those perf gains) we got another problem: unbounded page reconstruction which we had to solve separately.

I'm a VP on Databricks and former CEO of Neon. Happy to answer performance related or any other questions here.