What is extreme scale here? Like you've been able to fit over 1 PB of NVMe in 1U for years now (and before the AI inflation, prices were low enough that you could do it for like $100k IIRC). Are we talking 10s+ of PB? Who actually needs this scale?
At mere 100s of TB of data, I think you'll find that a normal database is far more likely to scale better. Most developers are... not as good as database developers at maintaining a database, and will not do a good job of it when they write their own inside of their application. They're often not even that good at using a database.
e.g.
> You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further
You can just do this with postgres. It works great. Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?
At mere 100s of TB of data, I think you'll find that a normal database is far more likely to scale better. Most developers are... not as good as database developers at maintaining a database, and will not do a good job of it when they write their own inside of their application. They're often not even that good at using a database.
e.g.
> You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further
You can just do this with postgres. It works great. Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?