How are people doing autoscaling clusters on AWS?
I just want to create a simple, single-AZ cluster with one master, and one autoscaling group for workers. Ideally when there are no jobs to run, the ASG scales down to zero.
(I have a Jenkins service that can spawn pods, and it works great on the single-master-only cluster with no workers. There's even room for it to spawn one pod alongside it on the master node. I just removed the dedicated taint annotation. But as soon as I schedule one more pod, I wish I had an ASG for worker nodes because I'm requesting more CPU and RAM than is available, so stuck waiting for the first pod to complete and resources to free up.)
If I had more nodes (or the possibility of more nodes), I'd likely put the dedicated taint back and give my Jenkins master server pod a tolerates annotation so that it can land there when the resources are available, or use a label selector in the pod spec to ensure it winds up on the stable master, or both. I think I know enough to put this combination together, but I am sure I can't be the first, and I don't want to miss the "golden path" if there is one, this thread seemed like a good place to ask about it.
This seems like it would be a really common configuration, but I haven't been able to find a lot of documentation or even blogs about setting up ASGs this way.
I found kube-aws[1], which seems to be the only K8S orchestrator that actually mentions creating AutoScaling Groups for worker nodes in the setup docs (??). I found cluster-autoscaler[2] which appears to be the piece you need to get the cluster to scale up an ASG when unsatisfied K8S resource requests have left pending scheduled pods waiting, or scale down when nodes are idle for too long. I'm totally mystified why this does not appear to be a supported configuration of Tectonic or Mantl or Stackpoint, or... what's your favorite K8s orchestrator and how does it handle this?
[1]: https://github.com/kubernetes-incubator/kube-aws
[2]: https://github.com/kubernetes/autoscaler/tree/master/cluster...
I just want to create a simple, single-AZ cluster with one master, and one autoscaling group for workers. Ideally when there are no jobs to run, the ASG scales down to zero.
(I have a Jenkins service that can spawn pods, and it works great on the single-master-only cluster with no workers. There's even room for it to spawn one pod alongside it on the master node. I just removed the dedicated taint annotation. But as soon as I schedule one more pod, I wish I had an ASG for worker nodes because I'm requesting more CPU and RAM than is available, so stuck waiting for the first pod to complete and resources to free up.)
If I had more nodes (or the possibility of more nodes), I'd likely put the dedicated taint back and give my Jenkins master server pod a tolerates annotation so that it can land there when the resources are available, or use a label selector in the pod spec to ensure it winds up on the stable master, or both. I think I know enough to put this combination together, but I am sure I can't be the first, and I don't want to miss the "golden path" if there is one, this thread seemed like a good place to ask about it.
This seems like it would be a really common configuration, but I haven't been able to find a lot of documentation or even blogs about setting up ASGs this way.
I found kube-aws[1], which seems to be the only K8S orchestrator that actually mentions creating AutoScaling Groups for worker nodes in the setup docs (??). I found cluster-autoscaler[2] which appears to be the piece you need to get the cluster to scale up an ASG when unsatisfied K8S resource requests have left pending scheduled pods waiting, or scale down when nodes are idle for too long. I'm totally mystified why this does not appear to be a supported configuration of Tectonic or Mantl or Stackpoint, or... what's your favorite K8s orchestrator and how does it handle this?
[1]: https://github.com/kubernetes-incubator/kube-aws
[2]: https://github.com/kubernetes/autoscaler/tree/master/cluster...