Comment by rodarmor

General-purpose deduplication sounds good in theory but tends not to work out in practice. IPFS uses a rolling hash with variable-sized pieces, in an attempt to deduplicate data rysnc-style. However, in practice, it doesn't actually make a difference, and adds complexity for no reason.