I'm approaching this problem via recursive task decomposition [0] mostly for coding but once that is polished, I can imagine it generalizing to other workloads. One idea I've been meaning to explore for a while is using the principles from Erlang/OTP [1] and mapping it onto multi-agent systems. Some nodes are supervisor trees that coordinate and handle restarts, gracefully degrade, are easy to observe + steer.
[0] https://github.com/adagradschool/scope [1] https://erlang.org/download/armstrong_thesis_2003.pdf