hagen1778

Most time series databases are columnar in some way. It is just because sotring and compressing timestamps and values as separate columns is way more efficient. So it is not a trait on its own, unless you compare yourself with Postgres.

> a) there's no per-series inverted index and labels are parquet columns so memory is not bounded by cardinality

Lack of inverted index is trade-off, in the same way as having that index. Default per-label inverted index in Prometheus, Mimir, VictoriaMetrics is convenient for users - requires no configuration from user side, all read queries, despite which time series selector was used, are as fast as they could be. Although, VictoriaMetrics doesn't store its whole index in-memory. It only stores what's "hot" now.

If you don't index all the labels, then you either asking user to choose what matters most, or intelligently decides what should be indexed. Either of these choice can take you to a corner case when query speed will be negatively affected despite having enough resources to burn.

> c) columnar store helps with faster data scanning by aggressively pruning and filtering data out

Columnar layout is not enough. Faster reads are achieved by reducing random seeks. A sequential read of a big chunk of data will always outperform multiple reads that pull the same data but from many places.

The same data can be stored in a columnar database with and without preserving data-locality (a trait that helps doing sequential reads). Let's talk time series. If in columnar database I'd decide to sort the columns by some common label and not by "timestamps" - data locality will be lost for typical queries that request "last 30min" of data.

I work at VictoriaMetrics.

Just to clarify: VictoriaMetrics doesn't use bots for HN or for any other media for promotion.

I don't know the person who you responded to. Most of the activity you see is coming from community members who genuinely use the project or from the core engineering team trying to answer user's questions or address misunderstandings.

> never responding to any comments

Could you please share examples like this? I can't say for community members, but our internal policy for engineers is very much focused on great support. You can check our slack/github to see that every question is answered and well explained.

I am curious to see more tests on the reading path. The article mentions matching 500 series over 6h window with 1m step - and it takes 2s for warmed caches. That doesn't sound good at all.

Especially nowadays, when metrics from k8s ramping up churn rate to hundreds of thousands and millions series.

What do you use instead of Prometheus?

Disclaimer: I am affiliated with VictoriaMetrics.

> The test setup is divergent from the real world (single node k8s cluster)

The setup was chosen to simplify the suite, so it can be easily run anywhere. In real world, log collectors are mostly deployed as deamonsets and this is what was tested during the benchmark. The vlagent was initially developed to run as a deployment, though. So I don't think changing setup will affect its performance.

> Their product is #1 in every metric they measured, but is also missing features from those compared against. Will those features change the results?

Depending on what features will be involved into the testing. In the benchmark, all collectors are doing the same job: collecting logs, parsing JSONs, shipping log records. So they are even in used features. Of course, the #1 product is missing features for log transformations, but these features aren't used during testing and shouldn't affect performance of other log collectors.

The bottom line of the post has "Should I switch" section explaining what's missing yet in product #1 for transparency.

Just to add, VictoriaMetrics covers all 3 signals:

- VictoriaMetrics for metrics. With Prometheus API support, so it integrates with Grafana using Prometheus datasource. It has its own Grafana datasource with extra functionality too.

- VictoriaLogs for logs. Integrates natively with Grafana using VictoriaLogs datasource.

- VictoriaTraces for traces. With Jaeger API support, so it intergrates with Grafana using Jaeger datasource.

All 3 solutions support alerting, managed by same team, are Apache2 licensed, are focused on resource efficiency and simiplicity.

There are plenty of ways to scale Prometheus:

- Thanos

- Mimir

- VictoriaMetrics

All of them provide a way to scale monitoring to insane numbers. The difference is in architecture, maintainability and performance. But make your own choices here.

Before, I remember there was m3db from Uber. But the project seems pretty dead now.

And there was a Cortex project, mostly maintaned by GrafanaLabs. But at some point they forked Cortex and named it Mimir. And Cortex is now maintained by Amazon and, as I undersand, is powering Amazon Managed Prometheus. However, I would avoid using Cortex ecaxctly because it is now maintained by Amazon.

My understanding is that with Prometheus+Grafana, and the rest of their stack, you can achieve the same functionality as Datadog (or even more) at much lower costs. But, it requires engineering time to set up these tools, monitor them, build dashboards and alerts. Build an observability platform at home, in other words.

But what about other open source solutions that already trying very hard to become an out-of-box solution for observability? Things like Netdata, Hyperdx, Coroot, etc. are already platforms for all telemetry signals, with fancy UIs and a lot of presets. Why people don't use them instead of Datadog?

Ofc you need to monitor your monitoring, because you run it. Datadog runs their own systems and monitors them, that's why they charge you so much. I barely can imagine a criticial piece of software that I need to run and not monitor it in the same time.