I'm fairly amazed by some of the choices - in particular the single huge user table that is hit on every profile page view to map url to shard. I'm guessing it has hot backups and similar - but wow - talk about a single point of failure. I guess having it like that actually makes it easier. If it fails, it should be trivial to get it back up quickly on a backup machine.
The whole setup seems a lot simpler than nearly every 'web scale' solution I've heard discussed.
EDIT: I'm not saying that is a bad thing - it's a good thing. Most of the articles around scaling on HN often deal with very complex Cassandra style technical solutions. This article is the opposite with a very 'just do it the simplest way' feel to it. One big mysql table for all users (probably with some memcache in front) is the sort of thing you can put together in an hour or two - yet it is the very core of their system.
I (and I bet a lot of other engineers reading HN) try to focus too much on perfect implementations - if someone asked me to design the shard look-up for something as heavily hit as this, I'd probably reach for a distributed in-memory DB with broadcasted updates - and I'd probably be wrong to do so. The mysql/memcache approach is just going to be simpler to keep running.
The whole setup seems a lot simpler than nearly every 'web scale' solution I've heard discussed.
EDIT: I'm not saying that is a bad thing - it's a good thing. Most of the articles around scaling on HN often deal with very complex Cassandra style technical solutions. This article is the opposite with a very 'just do it the simplest way' feel to it. One big mysql table for all users (probably with some memcache in front) is the sort of thing you can put together in an hour or two - yet it is the very core of their system.
I (and I bet a lot of other engineers reading HN) try to focus too much on perfect implementations - if someone asked me to design the shard look-up for something as heavily hit as this, I'd probably reach for a distributed in-memory DB with broadcasted updates - and I'd probably be wrong to do so. The mysql/memcache approach is just going to be simpler to keep running.