One thing missing here is to avoid synchronous communication. Sync comms tie client state to server state; if the server fails, the client will be responsible for handling it.
If you use queue-based services your clients can 'fire and forget', and then your error handling logic can be encapsulated by the queue/ consumers.
This means that if you deploy broken code rather than a cascading failure across all of your systems you just have a queue backup. Queue backups are also really easy to monitor, and make a great smoke-signal alert.
The other way to go, for sync comms, would be circuit breakers.
My current project uses queue-based communications exclusively and it's great. I have retry-queues, which use over-provisioned compute, and a dead-letter for manually investigating messages that caused persistent failures.
Isolation of state is probably the #1 suggestion I have for building scalable, resilient, self-healing services.
100% agree with and would echo the content in the article, otherwise.
edit: Also, idempotency. It's worth taking the time to write idempotent services.
If you use queue-based services your clients can 'fire and forget', and then your error handling logic can be encapsulated by the queue/ consumers.
This means that if you deploy broken code rather than a cascading failure across all of your systems you just have a queue backup. Queue backups are also really easy to monitor, and make a great smoke-signal alert.
The other way to go, for sync comms, would be circuit breakers.
My current project uses queue-based communications exclusively and it's great. I have retry-queues, which use over-provisioned compute, and a dead-letter for manually investigating messages that caused persistent failures.
Isolation of state is probably the #1 suggestion I have for building scalable, resilient, self-healing services.
100% agree with and would echo the content in the article, otherwise.
edit: Also, idempotency. It's worth taking the time to write idempotent services.