mattlord

So you don't understand how something works. That's fine. But to then say the article and/or tech are BS is... a choice.

This work has been and is being used by some of the largest sites / apps in the world including Uber, Slack, GitHub, Square... But sure, "it's BS, super custom, and irrelevant". Gee, yer super smart! Thank you for the amazing insights. 5 stars.

You are correct about resuming. If you do an initial VDiff and then resume that same VDiff say 1 month later it would only diff rows with a higher PK value.

But there's also nothing stopping you from doing a new VDiff to cover all data at that later point in time.

Blog post author here. I'm happy to answer any related questions you may have.

> If I were to do it in PostgreSQL 14 natively without any other third party plugins, I can get help of Table Partitioning (say root server) + Tables on Foreign Servers (say leaf servers) through FDW. Parallel scans and JOINs are allowed. Also, PostgreSQL 15 (in beta now) will allow parallel commits to foreign server tables through FDW.

I wish you all the best in operating that at scale and providing a reliable general purpose PostgreSQL DBaaS (even if only internally) around it. :-) PostgreSQL is great, not going to argue otherwise -- every solution has tradeoffs and if the tradeoffs of one solution fit your usage best then go with it. If you need to scale out and manage a large cluster of PostgreSQL instances as if it was a single logical database then I'd recommend something that's in a similar space as Vitess and PlanetScale -- Citus and Citus on Azure respectively. At the hand waving level it's always easy to "do this, and that", but it's a whole other matter when you develop a general purpose production-grade product that anyone can use and operate at virtually infinite scale. :-)

The Expensify test was targeted at demonstrating the upper limits of SQLite itself. The hardware used: > 1TB of DDR4 RAM > 3TB of NVME SSD storage > 192 physical 2.7GHz cores (384 with hyperthreading)

And a very simple read-only query w/o overlapping rows, with the full dataset already loaded in memory: > The query itself just sums a range of 10 rows randomly located inside 10B row (447GB) two column database of random integers, one of which is indexed and the other not. This means that the test is fully cached in RAM, and the database is precached prior to the test. No writes are done during this test, only reads.

All that is good and fine for that purpose -- to demonstrate the limits of what SQLite can do in scaling UP.

The purpose of the PlanetScale post was demonstrating linear scalability by scaling OUT using a realistic workload (TPC-C is a warehouse / ecommerce site simulation) on small commodity VMs (ec2 isntances).

As noted, you're comparing two very different tests that had very different goals.

I doubt that anyone would claim their billing metrics are perfect. If you find some specific workload that's actually cheaper on another serverless database offering then we'd love to hear about it (we strive for transparent, generous pricing). If you don't think that CPU usage based pricing — which is typical for serverless offerings and e.g. is what Aurora serverless uses in Aurora Capacity Units (ACUs) — is charging you for reads of cached data then I've got some bad news for you. :-) You're almost certainly being charged for reading the "row" from the network, write-ahead-logging for it and other ACID/MVCC related overhead, writing it to block device, reading it from the block device, reading it from memory, writing it to memory, sorting and comparing [pieces of] it, and writing it back to the network — all of these things take CPU cycles. I find this argument to be entirely missing the point.

Pointing out that surely Amazon would like to keep their patch set to a minimum (there's a high cost in maintaining custom patches as you upgrade MySQL) is in no way implying that their patch set is small. Minimal means the minimum required for what you need, rather than being some point of pride.

I'm certainly not on here bashing any other offerings. Between the two of us, I only see one person trolling / bashing. :-) With that, I will leave you to your opinions which you are of course free to have. Best of luck.

This is at least the second time you've made this exact same comment on an article. So in an attempt to respond to what I will assume are arguments made in good faith...

A database stores and serves requested data. Pricing is based on your usage of this service — how much data are you requesting it to serve in this case. It does not matter if a piece of data is cached or not, it's a database service not a block device service — and certainly not one with an independent cache in front of it that is somehow magically free to operate. Are you insinuating that e.g. Redis should always be free because its keys/records are in memory? I'm sorry, but I fail to see the logic in your argument here.

Of course ELT jobs consume resources. You're using the service, and you're paying based on your usage. This is simply how serverless / usage based models work. Again, I fail to follow the logic here.

innodb_rows_read is a count of the rows that MySQL (the query execution layer) reads from InnoDB (via the storage engine API). This does not count reads from internal / system / data dictionary tables as that is covered by innodb_system_rows_read. It's not "a bug" that it is what it is — InnoDB internally reads (b+ tree) pages and index records (as it uses index organized tables), and that's what happens with Index Condition pushdowns, it can apply query predicates directly as it's examining the index leaf nodes; it's only when pushing results back up to MySQL through the storage engine interface that these InnoDB records get converted to MySQL's generic row format (which includes going from InnoDB's arch independent big endian format to system/little endian format). This metric is telling you exactly what it should (not a bug) — how many rows does MySQL read from InnoDB — there are other handler and InnoDB specific internal metrics that you can use if you want other numbers (e.g. information_schema.innodb_metrics). JFG's blog post was (correctly) noting that if you as an system operator are trying to calculate the full cost of a query then you cannot rely solely on how many rows MySQL reads from InnoDB but instead look at how much work InnoDB does (which is much more challenging as you have b-tree traversal and maintenance, MVCC overhead, prefretching, etc involved) — and ultimately how many system resources are used in total (I/O amplification being one factor with InnoDB's index organized tables and update-in-place model). So parsing and optimizing costs, how many bytes are read from disk, pages from memory, was a temp table used, was a sort file used, etc. Again, this innodb_rows_read metric's intended behavior/meaning is NOT a bug and this metric will always favor the user for usage based billing. Your insinuation that this is somehow buggy and cannot be trusted as a metric so beware... here I would say that you are simply mistaken. This was an intentional decision made to offer simplicity, transparency, and to benefit users (we also go through great efforts to subtract rows read due to internal operations, just as MySQL itself does).

Hopefully this helps to allay your concerns.