And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says.
There is a provision in most file systems to use links (symlinks,
hardlinks, etc.). Links can cause cycles in the file tree, which
would result in a traversal algorithm going into an infinite loop.
To prevent this from happening, we ignore links in the file tree
during traversal. We note that the algorithms we propose in
the paper will duplicate effort proportional to the number of
hardlinks. However, in real world production systems, such as
in LANL (and others), for simplicity, the parallel filesystems
are generally not POSIX compliant, that is, they do not use
hard links, inodes, and symlinks. So, our assumption holds.
The reason this cp took such large amounts of time was the desire to preserve hardlinks and the resize of the hashtable used to track the device and inode of the source and destination files.
There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To prevent this from happening, we ignore links in the file tree during traversal. We note that the algorithms we propose in the paper will duplicate effort proportional to the number of hardlinks. However, in real world production systems, such as in LANL (and others), for simplicity, the parallel filesystems are generally not POSIX compliant, that is, they do not use hard links, inodes, and symlinks. So, our assumption holds.
The reason this cp took such large amounts of time was the desire to preserve hardlinks and the resize of the hashtable used to track the device and inode of the source and destination files.