powturbo
Joined 12 years ago
Joined 12 years ago
Well, a correct benchmarking is not done with special data, but with datasets that represent a large set of distributions. Such datasets are for ex. einwik8/9 for text, silesia for a mixed dataset. As a corner case example, RLE-compressible data is not representative for benchmarking compression libraries.
If you provide a link for a dataset 10-100MB, I can verify your claims, because I'm not aware of a dataset where igzip is 2 times faster than libdeflate. In TurboBench there is no I/O or other overhead involved, additionally it's single threaded. It's also possible that you're comparing two different CLI programs, one (igzip) I/O optimized and the other as a simple CLI.
See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,...
Number of pages = 1178
Average length = 84886 bytes
CPU: AMD 7840HS 3.8-5.1GHZ
Download TurboBench from Releases [2]
Here Some Benchmarks:
- https://github.com/zlib-ng/zlib-ng/issues/1486
- https://github.com/powturbo/TurboBench/issues/43
- 20-30x faster than the standard library
- Benchmarks faster than any other C base64 library
- Fastest implementation of AVX, AVX2, and AVX512 base64 encoding
- No other dependencies
Tired of misleading benchmarks with I/O overhead, cache effects, and CPU throttling? TurboBench delivers high-precision, zero-overhead results you can actually trust.
Why TurboBench Stands Out The only benchmark that includes 100+ popular, latest, and fastest compressors (LZ77, Rolz, BWT, Context Mixing, Entropy Coders, and more). 100% in-memory testing — no disk I/O to skew your results. Supports multiple files, recursive directories, and smart concatenation of small files into multiblock tests. Built to avoid common pitfalls like cache scenarios and throttling found in other tools. Unlock peak performance — TurboBench reduces throttling by intelligently controlling the CPU frequency Automatic sorting, updating, and merging of result files. A nightly github release is created automatically whenever submodules change. Creates a separate result file ([email protected]) for each compressor set TurboBench rank aggregation crowns the top compression libraries with — see who's king!