> How does data placement work?
Each object bigger than some size is split in small parts, these parts are linked to the metadata object with all information, such as name, bucket, size, date, checksum, etc.
All data is split in server groups - each group is at least 3 mirrored servers with no more than 5TB of data to make system flexible. Server groups can be added to increase system capacity or removed to decrease.
> How is data checked for correctness?
With checksum. Once the data is uploaded by user he will receive its checksum and must compared it with local checksum to make sure that it was correctly transfered and stored. The same checksum is used to ensure server-side data correctness.
> How do you do listings at scale?
There is a trick - we support only one delimiter (/), this means that we can use very simple listing algorithm which scales very easy.
> During hardware failures, do you still make any durability or availability guarantees?
Yes, all data is split in server groups by 3 servers each. If one of 3 servers will fail, this group will still running like nothing happened, some running requests may fail though. If 2 servers will fail at the same time, then this group and all data in it will be put in read-only mode to avoid any possible data damage.
> How do you handle hot content?
It is cached in RAM by OS, we do not perform any additional measures. OS does a pretty good job.
> How do you resolve conflicts (eg concurrent writes)?
Some conflicts are resolved by the software if possible. Unrecoverable conflicts are returned back to user with HTTP 400, 500 errors to make him know that something is wrong and he must run request again.
For concurrent writes we use simple rule - the last one wins.
> How is data checked for correctness? With checksum. Once the data is uploaded by user he will receive its checksum and must compared it with local checksum to make sure that it was correctly transfered and stored. The same checksum is used to ensure server-side data correctness.
> How do you do listings at scale? There is a trick - we support only one delimiter (/), this means that we can use very simple listing algorithm which scales very easy.
> During hardware failures, do you still make any durability or availability guarantees? Yes, all data is split in server groups by 3 servers each. If one of 3 servers will fail, this group will still running like nothing happened, some running requests may fail though. If 2 servers will fail at the same time, then this group and all data in it will be put in read-only mode to avoid any possible data damage.
> How do you handle hot content? It is cached in RAM by OS, we do not perform any additional measures. OS does a pretty good job.
> How do you resolve conflicts (eg concurrent writes)? Some conflicts are resolved by the software if possible. Unrecoverable conflicts are returned back to user with HTTP 400, 500 errors to make him know that something is wrong and he must run request again. For concurrent writes we use simple rule - the last one wins.