My current company is split... maybe 75/25 (at this point) between Kubernetes and a bespoke, Ansible-driven deployment system that manually runs Docker containers on nodes in an AWS ASG and will take care of deregistering/reregistering the nodes with the ALB while the containers on a given node are getting futzed with. The Ansible method works remarkably well for it's age, but the big thing I use to convince teams to move to Kubernetes is that we can take your peak deploy times from, say, a couple hours down to a few minutes, and you can autoscale far faster and more efficiently than you can with CPU-based scaling on an ASG.
From service teams that have done the migrations, the things I hear consistently though are:
- when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!)
- they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax.
[Both of those things are very much fixable]
From service teams that have done the migrations, the things I hear consistently though are:
- when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!)
- they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax.
[Both of those things are very much fixable]