I used to work at Second Life. Their player inventory and monetary transaction system was in MySQL. Every single item a player had or made was a record in this database. If you’ve never spent time in SL then the scope of this is probably a lot bigger than you imagine. As an active creator in SL I had 10s of thousands of records in there myself for inventory, and lots more activity for financial transactions and history. The database was distributed into somewhere between 30 and 50 servers, each of which had a read only replica. User inventory was divided amongst these shards. Any transactions that could execute on the read only replica did, and transactions that needed to be placed on the main server for a user executed there.
There were always concerns about the latency involved in the replication. Whenever I did large scale batch operations that needed to write lots of records in master servers, I would monitor the replication queue length and suspend operations when it got too far behind to keep things sane and healthy.
The biggest failure in this architecture was sharing sql queries as opposed to sharing service calls. This is something that was being fixed as I left. Switching it to work with service calls gives more flexibility in the underlying implementation and opens more options for improving the system.
For most kinds of large systems I’d recommend thinking about the architecture in terms of distributed Actors hosted in something like DAPR, Orleans, Akka, or Erlang/OTP and then the technical details of the underlying database become less of a constraint. It grants you flexibility for your underlying storage that you would not otherwise have.
Fun side fact: Bos, the CTO of meta, was a key person working on making all the magic happen there back then.
There were always concerns about the latency involved in the replication. Whenever I did large scale batch operations that needed to write lots of records in master servers, I would monitor the replication queue length and suspend operations when it got too far behind to keep things sane and healthy.
The biggest failure in this architecture was sharing sql queries as opposed to sharing service calls. This is something that was being fixed as I left. Switching it to work with service calls gives more flexibility in the underlying implementation and opens more options for improving the system.
For most kinds of large systems I’d recommend thinking about the architecture in terms of distributed Actors hosted in something like DAPR, Orleans, Akka, or Erlang/OTP and then the technical details of the underlying database become less of a constraint. It grants you flexibility for your underlying storage that you would not otherwise have.
Fun side fact: Bos, the CTO of meta, was a key person working on making all the magic happen there back then.