On a recent trip, while stuck on spotty hotel Wi-Fi and roaming 5G, I got fed up with how much bandwidth commands like rsync and nix-copy-closure waste.
rsync's delta transfer compares each file only with the file of the same name on the destination (or, with --fuzzy, a file in the same directory), so renamed, moved and copied files are sent in full.
nix-copy-closure can be worse: a rebuilt package gets a new store path, so the whole package is sent again, even when most of its contents haven't changed.
So I created hashsync, which runs a command over SSH and deduplicates the command's output, input, or both against an index of the receiving host's files.
As an example, for my last NixOS update, nix-copy-closure would have sent 1.7 GiB with --gzip, but hashsync took it down to 0.7 GiB instead. On some of my backups, I've seen up to 2x-10x less data and time to transfer compared to rsync with compression.
Two things make hashsync unique:
1. It searches at every byte offset of the stream, whatever its format, for blocks of the indexed files. In effect, it works like rsync's algorithm, but across every indexed file.
2. It's a transparent pipe, so it works with rsync, tar, or anything else you'd send through ssh.
See the repo's README for more info.
Feel free to try it out and let me know if you have any questions!
rsync's delta transfer compares each file only with the file of the same name on the destination (or, with --fuzzy, a file in the same directory), so renamed, moved and copied files are sent in full.
nix-copy-closure can be worse: a rebuilt package gets a new store path, so the whole package is sent again, even when most of its contents haven't changed.
So I created hashsync, which runs a command over SSH and deduplicates the command's output, input, or both against an index of the receiving host's files.
As an example, for my last NixOS update, nix-copy-closure would have sent 1.7 GiB with --gzip, but hashsync took it down to 0.7 GiB instead. On some of my backups, I've seen up to 2x-10x less data and time to transfer compared to rsync with compression.
Two things make hashsync unique:
1. It searches at every byte offset of the stream, whatever its format, for blocks of the indexed files. In effect, it works like rsync's algorithm, but across every indexed file.
2. It's a transparent pipe, so it works with rsync, tar, or anything else you'd send through ssh.
See the repo's README for more info.
Feel free to try it out and let me know if you have any questions!