_willmanning
Joined 7 years ago
Karma
8
Submissions
6
Member since
Sep 4
Recent submissions
Comment · 4 months ago
it would have been easier to just go to FNAC and buy Rocket League like a normal person :)
Comment · 1 year ago
Vs Apache Parquet: 100x faster random access, 10-20x faster scans, 5x faster writes, similar compression ratio
And nearly as fast as duckdb's native format on Clickbench when queried from duckdb
Comment · 1 year ago
I'll chime in (as a Vortex maintainer), that we are greatly indebted to Azim's work on FastLanes & ALP. Vortex heavily utilizes his work to get state-of-the-art performance.
I would add that Vortex doesn't have standalone Clickbench results. Azim is presumably referring to the duckdb-vortex results, which were run on an older version of duckdb (1.2) than the duckdb-parquet ones (1.3). We'll get those updated shortly; we just released a new version of Vortex & the duckdb extension. Meanwhile, I believe the DataFusion-Vortex vs DataFusion-Parquet speedups show substantial improvements across the board.
The folks over at TUM (who originally authored BtrBlocks) did a reasonable amount of micro-benchmarking of Vortex vs Parquet in their recent "Anyblox" paper for VLDB 2025: https://gienieczko.com/anyblox-paper
They essentially say in the paper that Vortex is much faster than the original BtrBlocks because it uses better encodings (specifically citing FastLanes & ALP).
I'm looking forward to seeing the FastLanes Clickbench results when they're ready, and Azim, we should work together to benchmark FastLanes against Vortex!
Comment · 2 years ago
Compression! Vortex can easily be 10x smaller than the equivalent Arrow representation (and decompresses very quickly into Arrow)
Comment · 2 years ago
Perhaps that verbiage is just confusing. "On-disk" sort of implies "file format" but could be more explicit.
That said, the immediate next line in the README perhaps clarifies a bit?
"Vortex is designed to be to columnar file formats what Apache DataFusion is to query engines (or, analogously, what LLVM + Clang are to compilers): a highly extensible & extremely fast framework for building a modern columnar file format, with a state-of-the-art, "batteries included" reference implementation."