Comment by ozkatz

Curious if you’ve had a chance to try lakeFS?

It was designed with large-scale environments in mind. I’m aware of several deployments managing hundreds of petabytes of data and billions of objects, which is why lakeFS does not use Git’s Merkle tree / directory tree approach.

Disclaimer: I’m one of the project’s co-creators.