james_cowling

I'm giving the first talk of the session and while I'll be talking about ideas we've actually built and tested in Convex, the point I want to get across is that I think the industry has gone in the wrong direction with exposing databases to clients and using an inexpressive declarative language to query them. Will be trying to avoid any shilling because I feel like this is a pretty important discussion to have.

Yes the readsets aren't based on lists of objects but rather the data ranges that a query touches, i.e., they don't suffer from phantom read anomalies.

If you were to attempt to fetch all items from an empty shopping cart, that query will automatically be invalidated if any item is added to the cart, even though there were no documents returned from the original query. Query intersection always works.

The aggregate example isn't a great one in Convex since it doesn't currently support built-in aggregates - a full-table aggregate involves a table scan and therefore the readset would be the entire index. We may add built-in aggregates that are incrementally computed if there's enough demand.

Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time.

Internally the very bottom of the stack on the hosted product is actually just a write ahead log on RDS and in the open source product it's just sqlite. We'll likely eventually build our own durable write ahead log, and have plenty of experience doing so, but this hasn't been needed so far.

Convex cofounder here. Initial focus has been low-latency OLTP database workloads, which is pretty tied to the user-facing end-to-end reactivity angle we're pushing. As a bonus feature we also have built-in strongly consistent full text search, but not vector search.

So far our answer for folks who want alternative storage/query engines is to use our streaming Airbyte source connector or write directly to Pinecone, Snowflake, etc. This should work great for most devs.

There are likely always going to be some developers who want to use a particular third party database in addition to Convex, but we plan to expand built-in support for most workloads over time so that Convex is a truly full-stack backend replacement.

(kmod was the founding engineer on the project and the guy who came up with the name!)

We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then!

(a bunch of the early MP folks work at Convex now)

We had a lot of requests for open sourcing MP but I don't think it would have been very useful. The code itself is interesting to look at but the system is designed for very large data sets, has a lot of moving parts to manage and configure, and would be very difficult to operate at anything below at least double-digit petabytes.

MP was designed for multiple exabytes across many data centers and geographic regions. A different system design (or just using S3) would be more appropriate for use at smaller scales.

We designed and built the system almost completely ground-up, all the way down to the disk scheduler, but yeah we probably took the term OSD from Ceph.

Yep that’s me! I have my full name on here so wasn’t hiding but good point to be clear about disclosing.