AdamProut

I'm curious why the test needed so many router hosts:

512 shards, each with one Postgres primary each on an r8g.16xlarge

480 Neki routers, each on its own 8xlarge instance

That's ~250K queries/sec per router which seems lowish for this type of workload? The routers won't be doing very much (parse query, route it to proper shard?).

yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).

I'll admit to not reading your code. It's hard to keep up with all the different databases launched in the last decade.

I have doubts that a global readwrite lock around a hashtable makes for a good general purpose storage system.

what kind of maintenance?

I think one of the maybe less talked about benefits of distributed SQL is support for nearly transparent rolling upgrades of the database with very little impact to a running workload. Spanner is best in class at this.

Agents should be driving up the quality of software if we focus them on finding and fixing bugs.

Very cool! pg_duckdb itself is missing fully integrated storage - it can query data stored externally (say in S3) in delta/iceberg formats, but it can't write out data in those formats via transactional writes to PG tables (insert\update\deletes). pg_mooncacke is one neat way of solving that problem. It lets you have a columnstore table in Postgres that can do both reads and writes as if it's any other PG table and have the storage format be an open format like delta/iceberg/etc with that data persisted to blob store (like most cloud DWs would do anyways).

Had a similar thought. Azure Postgres has something similar to pg_parquet (pg_azure_storage), but we're looking into replacing it with pg_duckdb assuming the extension continues to mature.

It would be great if the Postgres community could get behind one good opensource extension for the various columnstore data use cases (querying data stored in an open columnstore format - delta, iceberg, etc. being one of them). pg_duckdb seems to have the best chance at being the goto extension for this.

Checksums can detect a torn page, but not always repair them. It's likely a good part of the database page is gone (i.e., an amount of data that matches the disk / file system atomic write unit size is probably missing). Torn page writes are a pretty common scenario too, so databases need to be able to fully recover from them - not just detect them and report a corruption (ie., just pull the power plug from the machine during a heavy write workload and you're likely to get one - it doesn't require a solar ray to flip a bit :) ).

I'm not sure why you're getting downvoted. I think the pressure to appease Gartner usually starts when companies bring in CEOs whose primary background is enterprise sales. They tend to over value magic quadrant positioning (in my view).

Based on the writings of 2 Turing awards winners for work on databases; No MongoDB doesn't have ACID transactions.