We've been using CF for a few years as well. IMHO it is very complicated to manage and getting stuff working involves a lot of trial and error.
Also you end up waiting a lot. Waiting for things to spin up, waiting for things to become available, waiting for things to rollback, etc. On top of that the failure modes can be ugly and hard to figure out.
My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.
So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.
If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.
My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.
So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.
If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.