I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons.
One approach I can think of is to run a distributed document database and use that for storage. I don't need most features of such database products in the foreseeable future, so I fear that they will add operational overhead for not much benefit.
Another approach I can think of is to run my processing nodes against a network file system, and rely on that to do the replication.
Yet another approach I am considering is to use something like CopyCat to implement the file replication in my application code. Is this a good use case for CopyCat?
One approach I can think of is to run a distributed document database and use that for storage. I don't need most features of such database products in the foreseeable future, so I fear that they will add operational overhead for not much benefit.
Another approach I can think of is to run my processing nodes against a network file system, and rely on that to do the replication.
Yet another approach I am considering is to use something like CopyCat to implement the file replication in my application code. Is this a good use case for CopyCat?