Comment

Comment by fintler

I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems.

https://github.com/hpc/dcp

We got an IEEE paper out of it:

http://conferences.computer.org/sc/2012/papers/1000a015.pdf

A few people are continuing the concept to other tools -- that should be available at http://fileutils.io/ relatively soon.

We also had another tool written on top of https://github.com/hpc/libcircle that would gather metadata on a few hundred-million files in a few hours (we had to limit the speed so it wouldn't take down the filesystem). For a slimmed down version of that tool, take a look at https://github.com/hpc/libdftw

parent fintler 12 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton