abadid

AI agents are bad at joining large tables.

The hype doesn't always live up to reality.

More proof that it is really slow to access data in a database over JDBC.

Short summary: Trino uses JDBC in ways in which it was not designed and this can result in massive performance bottlenecks. Sometimes partitioning source data can mitigate the bottleneck.

A lot of people don't realize how slow it is to access data in bulk from a database using JDBC. JDBC was never designed for the use case that Trino uses it for.

This article summarizes research from my lab in collaboration with ByteDance published in CIDR (a computer science conference held in Amsterdam two weeks from now) on a new columnar format designed for ML workloads.

Why use PostgreSQL instead of columnar databases that are likely to perform way better for these types of analytical workloads?

I agree that in theory they could both co-exist for the reasons you state, but in practice I think it's unlikely a company that invests in a data fabric (which is largely a technology cost) is going to simultaneously invest in the incentives for the data product creators that are necessary for the data mesh not to become the wild west.

In general, whenever you need to perform a join (of multiple datasets), that ends the pipeline of local operations on a partition. Other operators as well that necessarily require data from other partitions end local pipelines. This is why linear scalability is not completely achieved in practice. Most interactions with data cannot be performed in a completely partitionable way.

Usually those other operations which force the local pipeline to end occur in a query plan prior to hitting any kind of tradeoff of doing too much in a local pipeline, since local pipelines are SO much faster than what happens when communication is required.