At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) without any disruption to user traffic.
To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.
We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.
To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.
We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.