Comment by gazpacho

Author here. You hit the nail on the head. It's multi-tenant dynamic data. It varies between tenants, even within a single tenant there are large variations across data streams and across time. We decide the layout per physical file we write out using a combination of heuristics and query statistics (we shred attributes that are heavily queried).