gazpacho

Upsie, I posted my first draft and an attempt to rewrite to make more constructive. Take that for what you will, sorry for any hostility in the first draft.

Hi, author here.

The article doesn't claim to have made some huge discovery, it tells the journey of making our product / system much better by applying well known techniques. We linked to similar things (ClickHouse's handling of semi structured data, Parquet's Variant type). There are others (e.g. BigQuery). I'm sure there's some system that did similar things before (if you have a reference I'd be interested in learning the history here) but all of the ones I mentioned above are from 2022+.

But besides that I think your comment misses the entire point of the article: how do you normalize semi-structured data that varies in schema vastly across tenants, within those tenants between those systems and drifts over time rather rapidly? There is no schema to pick and normalize to.

Hi, author here.

I agree this is not some huge discovery. We link ClickHouse's JSON type and Parquet Variant in the post, and Snowflake's VARIANT and BigQuery do similar things. We didn't mean to present it as a discovery. The post is about how we applied it to our system.

The part normalization doesn't answer for us, as guites said in this thread, is that there's no schema to normalize to. Every tenant sends different OpenTelemetry attributes, different services within a tenant send different ones, and they change from week to week. Picking a fixed set of columns is what we tried first (we call it static shredding in the post). So each Parquet file now picks its own columns from the data it contains, without any configuration.

I know in the database world we are often recycling or tweaking ideas from decades ago so if you know where these ideas for handling semi-structured data in columnar databases comes from I'd be interested to learn!

Thanks for giving the post a fair shot!

In our case it's more than that: the data is dynamically changing, constantly, across tenants, across their data streams and across time.

What we're talking about here is basically dynamically creating a table structure from semi-structured data. It's not about "we don't know what the right columns are", it's about "there are no universally right columns, but there are some better and some worse ones within the bounded confines of some finite amount of data".

This is a great question. The short answer is IO patterns: because the data is stored as Parquet in object storage just knowing that a certain page/row group/file has the data you are looking for is only half of the battle. If you then have to download MBs of JSON to then extract that one value from that one row things will still be slow. Shredding allows the data to be accessed independently, in addition to unlocking encoding (delta, ALP, etc.).

In practice we do both: we have key presence indexes and zone maps on shredded and unshredded values. Then we go an grab the data and if it's shredded it's going to be faster to grab even once we know where it is.

Author here. You hit the nail on the head. It's multi-tenant dynamic data. It varies between tenants, even within a single tenant there are large variations across data streams and across time. We decide the layout per physical file we write out using a combination of heuristics and query statistics (we shred attributes that are heavily queried).

Another +1 for ADBC

Destroy oil supply and there is a crisis in 12h. Destroy solar supply and there's a crisis in... 20 years? It'd actually be much sooner but the point is that it's much less urgent that oil.

Love this take!

Modern society is too narrow minded about what wealth means. To most people it means fancy watches, cars, homes, etc. To me wealth is about time and freedom. I can pick up the tab at dinner for a group of 12. I can afford to keep my 1972 Schwinn bicycle in tip top shape or my grandfather's jacket mended when it breaks. I can afford to rent forever if I want. I'll never work a job I don't like, I can just quit. I feel more wealthy this way than if I owned 3 lambos and had to work to make ends meet.

Having only read the abstract... the conclusion makes sense to me. I've operated under the assumption that volume is the most important factor for muscle growth as long as you're lifting something like 1/3 or more of your 1RM. So 12 reps with higher load or 25 reps with lower load are going to be similar volumes (or at least similar enough given the other factors that the two protocols give the same outcome).

This is what I've found after 15 years of working out and athletics. Think of it this way: doing the same thing over and over again is what is proven to lead to workplace injuries. Doing the same thing over and over again in the gym is no different.

I like to do a weight training as the consistent foundation, with a mix of heavy lifts, calisthenics, volume (bodybuilding) training and mobility training. Add in some yoga, rock climbing, biking, soccer. I feel this sort of mix balances movements out which helps with injury prevention and also makes sure you always have something active to do that you enjoy, which is definitely #1.