1) While building this system, I found that the LLM can sometimes generate unpredictable responses. For example, the LLM sometimes chooses to summarize the document even for a simple retrieval question. When using expensive LLM models, this mistake could result in 10x higher cost. In your case, the LLM could generate sub-tasks that incur significant operating overheads. Just curious if you're currently facing such issues and if you have plans to mitigate them.
2) The restart idea is neat! I often faced this scenario where only few sub-questions have some issues that need to be fixed. Tweaking them without re-running the whole pipeline seems like a useful feature in this case.
2) The restart idea is neat! I often faced this scenario where only few sub-questions have some issues that need to be fixed. Tweaking them without re-running the whole pipeline seems like a useful feature in this case.