I follow a similar approach and use multiple LLMs per task. The quality improvement is surprisingly large.
Lately I’ve been experimenting with adding an explicit reward function so the models optimize for measurable output quality.
This creates a generate, critique, revise loop where candidate answers compete for a higher score. It feels promising because it reduces the amount of handholding for every task. It is also more fun because part of the review process is embedded in the scoring function, which simplifies the review effort.
Lately I’ve been experimenting with adding an explicit reward function so the models optimize for measurable output quality.
This creates a generate, critique, revise loop where candidate answers compete for a higher score. It feels promising because it reduces the amount of handholding for every task. It is also more fun because part of the review process is embedded in the scoring function, which simplifies the review effort.