mwlon

Numerical data is full of rich patterns, but the general-purpose compressors we've historically used on them (e.g. snappy, gzip, zstd) are designed for unstructured, string-like data. Pcodec (or pco) is a new approach for numerical sequences that gets better compression ratio and decompression speed than alternatives. It usually improves compression ratio substantially, given the same compression time. Plus it's built to perform on all common CPU architectures, decompressing around 1-4GB/s.

You might have seen me post about Quantile Compression in previous years. Pco is its successor! Pco gets slightly better compression ratio, robustly handles more types of data, and (most importantly) decompresses much faster.

If you're interested in using it, there's a Rust API, Python (PyO3) API, and a CLI.

PancakeDB is an event ingestion solution, an important part of most tech companies' data stacks. Write to it one event at a time, and process billions at a time with the Spark connector or other tools.

I've released it under BSL so that any company can run it on their own servers for free.

It is a new startup I'm building.

It's a new type of database that can take in streaming data with very fast (~10ms) response times and output batch data with very fast throughput. To do that, it uses a new columnar file format and compression algorithm. Together, this makes its columnar files 30-50% smaller under most circumstances while decoding just as quickly. That means storage costs are lower and it's 30+% faster assuming the same network bandwidth is used to transfer the data for all columns. And this is a pessimistic scenario, since most queries have a `select column_0, column_1, ...` clause that PancakeDB can leverage better than Parquet, transferring only the exact columns needed!

You can find edge cases (e.g. very long strings of uniformly random bytes) where it's only a few % faster instead of 30%, but in every real-world-resembling scenario I've tried, the advantage is much greater.