Comment Comment by gopalv > does duckdb support querying custom storage formats?No, this is still parquet being loaded in.The "trick" is running through sort order combinations which result in maximum packing in a Dremel style data format (like Parquet).Here's the trick explained in enough detail to understand, but not enough to copy as-ishttps://cloud.google.com/blog/products/bigquery/inside-capac...Search for the section with"Capacitor builds an approximation model that takes into account all relevant factors and comes up with a reasonable solution."or a more detailed dive in the paper (by the favourite author of performance engineers, Daniel Lemire)https://arxiv.org/abs/1207.2189 parent gopalv 9 days ago
No, this is still parquet being loaded in.
The "trick" is running through sort order combinations which result in maximum packing in a Dremel style data format (like Parquet).
Here's the trick explained in enough detail to understand, but not enough to copy as-is
https://cloud.google.com/blog/products/bigquery/inside-capac...
Search for the section with
"Capacitor builds an approximation model that takes into account all relevant factors and comes up with a reasonable solution."
or a more detailed dive in the paper (by the favourite author of performance engineers, Daniel Lemire)
https://arxiv.org/abs/1207.2189