Comment by big_dinghy

Want to shout out an alternative Python open source workflow orchestration tool, Prefect https://www.prefect.io/

It has a server/UI component that you can deploy relatively easily on something like Kubernetes. It then makes it easy to configure your flows to run on varying amounts of compute resources.