pierrebarre

I’m not saying the access pattern isn’t real. I’m saying it largely doesn’t matter in practice.

If Git repos were routinely terabytes in size, this would be a much harder problem. But the vast majority of repos are <1GB, and at that scale caching dominates. Even with imperfect locality, you can cache enough of the working set that those reads don’t turn into s3 round trips.

In a low-latency environment, I haven’t found filesystem round trips themselves to be an issue for Git workloads either.

> i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs

That can literally be the stack, but the same argument could be made about object storage itself: it ultimately writes data to local disks, so why not write files there directly?

The value of object storage isn’t just its API. It’s the durability, distribution, failure handling, and capacity behind that API. ZeroFS lets POSIX applications use those properties. Its server and cache are replaceable and the bucket remains the source of truth.

If one machine and its disks are enough for your data and reliability needs, then yes, a local filesystem is simpler. ZeroFS isn’t intended to improve that case.

Yes, I think it’s worth benchmarking.

zerofs mount has durable fsync and fcntl byte-range locking.

Inotify works for changes made through the local mount. It won’t report changes made through another client, though.

Small writes operate on 32 KiB extents. Partial overwrites are read-modify-write, but the FUSE writeback cache helps combine them, and the resulting extents are packed into segments rather than producing one S3 PUT per write.

I’d be interested in the results if you try it.

Hardlinks work as you’d expect: multiple paths point to the same inode and data, so a write through one path is visible through the others. Two separate files with the same contents are stored separately.

There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality.

If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.

Thanks!

stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached.

I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata.

The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.

Author here. Thanks for posting this!

It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.