If you’re already in the Kubernetes system, Argo Workflows has either capabilities designed around what you are describing or can be built using the templates supported (container, script, resource). If you’re not on Kubernetes, then Argo Workflows is not worth it on its own because it does demand expertise there to wield it effectively.
Someone suggested Temporal below and that’s a good suggestion too if you’re fine with a managed service.
Someone suggested Temporal below and that’s a good suggestion too if you’re fine with a managed service.