Recent submissions

Comment · 1 month ago
TurboBench: The Ultimate Data Compression Benchmark 100+ Codecs. The Powerhouse in a Single Executable — The Compression Lie Detector

Tired of misleading benchmarks with I/O overhead, cache effects, and CPU throttling? TurboBench delivers high-precision, zero-overhead results you can actually trust.

Why TurboBench Stands Out The only benchmark that includes 100+ popular, latest, and fastest compressors (LZ77, Rolz, BWT, Context Mixing, Entropy Coders, and more). 100% in-memory testing — no disk I/O to skew your results. Supports multiple files, recursive directories, and smart concatenation of small files into multiblock tests. Built to avoid common pitfalls like cache scenarios and throttling found in other tools. Unlock peak performance — TurboBench reduces throttling by intelligently controlling the CPU frequency Automatic sorting, updating, and merging of result files. A nightly github release is created automatically whenever submodules change. Creates a separate result file ([email protected]) for each compressor set TurboBench rank aggregation crowns the top compression libraries with — see who's king!

Comment · 3 years ago
Comment · 3 years ago
Thank for your analyse. It seems it's a corner case, benchmarking a base64 encoded random file is not a typical case. igzip has no more advantage above libdeflate. It has only fast compression at the levels 0,1,2 but with mediocre compression ratio.
Comment · 3 years ago
EDIT: I've seen the file you're referencing is 4Gi-Base64. This file is not very compressible (75% with gzip). It's possible that igzip is simply storing the file or some parts of it without compression. This explain why it can be faster that libdeflate, because in this case igzip is using memcpy at decompression.
Comment · 3 years ago
Sorry, a digit was missing, but now https://github.com/powturbo/TurboBench/issues/43

Well, a correct benchmarking is not done with special data, but with datasets that represent a large set of distributions. Such datasets are for ex. einwik8/9 for text, silesia for a mixed dataset. As a corner case example, RLE-compressible data is not representative for benchmarking compression libraries.

If you provide a link for a dataset 10-100MB, I can verify your claims, because I'm not aware of a dataset where igzip is 2 times faster than libdeflate. In TurboBench there is no I/O or other overhead involved, additionally it's single threaded. It's also possible that you're comparing two different CLI programs, one (igzip) I/O optimized and the other as a simple CLI.

Comment · 3 years ago
Comment · 3 years ago
file html8 : 100MB random html pages from a 1m Alexa Top sites corpus.

Number of pages = 1178

Average length = 84886 bytes

CPU: AMD 7840HS 3.8-5.1GHZ

Comment · 3 years ago
Comment · 3 years ago

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton