Comment

Comment by hagen1778

Most time series databases are columnar in some way. It is just because sotring and compressing timestamps and values as separate columns is way more efficient. So it is not a trait on its own, unless you compare yourself with Postgres.

> a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality

Lack of inverted index is trade-off, in the same way as having that index. Default per-label inverted index in Prometheus, Mimir, VictoriaMetrics is convenient for users - requires no configuration from user side, all read queries, despite which time series selector was used, are as fast as they could be. Although, VictoriaMetrics doesn't store its whole index in-memory. It only stores what's "hot" now.

If you don't index all the labels, then you either asking user to choose what matters most, or intelligently decides what should be indexed. Either of these choice can take you to a corner case when query speed will be negatively affected despite having enough resources to burn.

> c) columnar store helps with faster data scanning by aggressively pruning and filtering data out

Columnar layout is not enough. Faster reads are achieved by reducing random seeks. A sequential read of a big chunk of data will always outperform multiple reads that pull the same data but from many places.

The same data can be stored in a columnar database with and without preserving data-locality (a trait that helps doing sequential reads). Let's talk time series. If in columnar database I'd decide to sort the columns by some common label and not by "timestamps" - data locality will be lost for typical queries that request "last 30min" of data.

parent hagen1778 19 days ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton