felixhandte
Joined 12 years ago
Joined 12 years ago
We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people who, you know, didn't develop OpenZL. Your feedback is welcome!
- A white paper: https://arxiv.org/abs/2510.03203
- A blog post: https://engineering.fb.com/2025/10/06/developer-tools/openzl...
- The documentation website: https://openzl.org/
Code: https://github.com/facebook/openzl
Documentation: https://openzl.org/
White Paper: https://arxiv.org/abs/2510.03203
The first stage of Zstd does LZ77 matching, which transforms the input into "sequences", a series of instructions each of which describes some literals and one match. The literals component of the instruction says "the next L bytes of the message are these L bytes". The match component says "the next M bytes of the input are the M bytes N bytes ago".
If you want to construct a match between two strings that differ by one character, rather than saying "the next N bytes are the N bytes M bytes ago except for this one byte here which is X instead", Zstd just breaks it up into two sequences, the first part of the match, and then a single literal byte describing the changed byte, and then the rest of the match, which is described as being at offset 0. The encoding rules for Zstd define offset 0 to mean "the previously used match offset". This isn't as powerful as a Levenshtein edit, but it's a reasonable approximation.
The big advantage of this approach is that it doesn't require much additional machinery on the encoder or decoder, and thus remains very fast. Whereas implementing a whole edit description state machine would (I think) slow down decompression and especially compression enormously.
[0] https://datatracker.ietf.org/doc/html/rfc8878#name-repeat-of...
[ my public key: https://keybase.io/felix; my proof: https://keybase.io/felix/sigs/mjJ1GvUGKTRmYImkgCu8Z04zn4AetQ7MsGOGRCSMh-A ]