This is pretty interesting, because the effects of migrating from lz4 to zstd were:
- Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes)
- Average write operation duration decreased by 50% on our fullest machines
- No observable query performance effects
It seems like the better compression ratio and resulting reduced IO more than makes up for increased CPU compared to lz4. I wish they had mentioned the actual effect on CPU.
Compare to the recent thread "The LZ4 introduced in PostgreSQL 14 provides faster compression" [0] where the loudest voices were saying that zstd would not work due to increased CPU. This is a different layer (filesystem compression vs db compression), but this article represents an interesting data point in the conversation.
- Total storage usage reduced by ~21% (for our dataset, this is on the order of petabytes)
- Average write operation duration decreased by 50% on our fullest machines
- No observable query performance effects
It seems like the better compression ratio and resulting reduced IO more than makes up for increased CPU compared to lz4. I wish they had mentioned the actual effect on CPU.
Compare to the recent thread "The LZ4 introduced in PostgreSQL 14 provides faster compression" [0] where the loudest voices were saying that zstd would not work due to increased CPU. This is a different layer (filesystem compression vs db compression), but this article represents an interesting data point in the conversation.
[0]: https://news.ycombinator.com/item?id=29147656