isburmistrov

Joined 2 years ago

Karma

Submissions

Member since

Feb 16

Recent submissions

I agree that specialised DBs outperform a general-purpose OLAP database. The question is - what does outperform mean. In this area queries should not be actually ultra-fast, they should be reasonably fast to be comfortable. And so missing indexes for some attributes would be likely okay. Looking at https://clickhouse.com/blog/storing-log-data-in-clickhouse-f..., they added just bloom filters for columns. Which makes sense, but this is not a full-blown index, and likely it will yield reasonable results. But this all is theoretical, I haven't built such a solution by self (we're working on it now for in-house observability), so likely miss something that can only be discovered on practice.

Btw we use Victoria Metrics now at work. It works good, queries are fast. But we're forced to always think about cardinality, otherwise either performance or cost get hurt. This is okay for the predefined set of metrics & labels and works well, but it doesn't allow having deep explorations.

Comment · 2 years ago

Thanks for sharing!

> This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.

Really impressive results.

Comment · 2 years ago

And yet people use ClickHouse quite effectively for this very problem, see the comment here: https://news.ycombinator.com/item?id=39549218

There are also time-series databases out there that are OK with high cardinality: https://questdb.io/blog/2021/06/16/high-cardinality-time-ser...

Comment · 2 years ago

Thanks for sharing, didn't know about Motif

Comment · 2 years ago

Does it support "native sampling" described in the article? This is really important to keep the cost low.

Comment · 2 years ago

Never in the world I would have expected my post to cause the discussion about ODS flaws :D

Comment · 2 years ago

I think the storage architecture in ClickHouse and Elastic are very different. And I think compression in ClickHouse can be really damn good.

Don't have a good comparison at hands though, but a few random links on the topic:

https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-pl...

Comment · 2 years ago

Haha, I think this term is originated by Honeycomb team actually.

Why I prefer "wide event" over "structured log" as a term because it has this "wide" component that serves for 2 purposes: - it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it

But just a personal preference in the end.

Comment · 2 years ago

Yes, structured log exactly. Why I prefer "wide event" as a term because it has this "wide" component that serves for 2 purposes:

- it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it

But just a personal preference in the end.

Comment · 2 years ago

> Those are entirely different orders of magnitude both when it comes to size and how much usefulness you lose. In modern storage backends like Victoriametrics a counter gonna cost you around byte per metric per probe. And as you emit them periodically, that is essentially independent of incoming traffic

I thought this argument was about whether wide events can be used for metrics or metrics is a completely different concept. If we want to emulate metrics in events, we would also make them periodically independently of the traffic. Like emit them once in a while. Pretty much like Prometheus scraping works

Comment · 2 years ago

I don't think Meta's margins have something to do with this. Companies smaller in scale than Meta also have less data!

And yes, Scuba is in-memory, but it's not the requirement. Check this video out on how Honeycomb implemented their columnar storage: https://vimeo.com/331143124

Comment · 2 years ago

I think ClickHouse is becoming a default storage for observability nowdays: https://clickhouse.com/use-cases/logging-and-metrics

And there are quite a few solutions on top of it.

A couple of examples that seem to be interesting (however I didn't use them in real life):

https://coroot.com/ https://qryn.metrico.in/#/

Comment · 2 years ago

Columnar storage stores data very efficiently, too - because it compresses data of a similar nature (columns). Check e.g. ClickHouse on this matter: https://clickhouse.com/docs/en/about-us/distinctive-features, https://clickhouse.com/blog/working-with-time-series-data-an...

So I wouldn't say that events are "expensive" while metrics are "cheap" - both depend on the actual implementation, and events can be cheap too.

And so of course if you have to optimise things, you would need to drop some information you pass to the events, but you would need to do the same for metrics (reduce the number of metrics emitted, reduce the prometheus labels,...).

Comment · 2 years ago

Sampling can be smart, e.g. based on some field all events have (can be called traceId, haha).

Comment · 2 years ago

> and "stop sampling" is just a bizarre marketing angle

Wait, where did I mention stopping sampling? :) The opposite: the article is praising the native sampling Scuba has.