dashdoesdata

At StarTree, we've taken a novel approach to querying Iceberg tables: applying Apache Pinot-style indexes to improve performance, lower costs, and increase concurrency… without moving data into a separate system.

In our tests on a ~1TB dataset, this reduced query latency significantly:

* 500+ QPS with sub-second latency * Complex aggregations <650ms * ~5–37x faster than ClickHouse and ~4–17x faster than Trino in the same setup

We welcome comments and input from this community!

Sadly, the economics of this will never work out.

I ride BART and Caltrain multiple times a week. Trains are delightful. And so are planes.

It typically costs about $100, and takes an hour to fly from SF to LA. The train will be slower and almost certainly more expensive. Where is the benefit?

Long distance trains work in Europe because they are supported by a rich network of public transit options stemming from city centers, and a population that uses transit frequently.

But California public transit is a mess. When you rock up at LA Union Central and want to get out to say, Newport Beach, it's another three hours of navigating buses and the metrolink.

The money would have been far better spent integrating agencies and building out high speed rail in our urban areas. BART should be running out to Sacto, Stockton, Silicon Valley, and Marin. Requisition the Amtrak line and build a highspeed rail from San Diego to Santa Barbara - now that'd be something.

Interesting post from Rajat Rana on Meesho's technology blog. Meesho is a huge Indian e-commerce platform. He shares how, and why, they rebuilt their real-time analytics platform on Apache Pinot to support a wider variety of sophisticated features — such as identity merging, deep insights, funnel analysis, and complex user journeys.

(I work at StarTree - a real-time analytics platform powered by Pinot - we helped them implement this)