Comment

Comment by torginus

Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.

So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.

It didn't work, or the gains were so marginal that I ended up scrapping the approach.

So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.

parent torginus 1 month ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton