Interestingly the cause of the most severe part of their downtime seems to still be unknown to them:
As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.
We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.
To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?
As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.
We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.
To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?