You could timeout. You could trade them off dynamically.
I.e. you get 3 replies. 80% confidence. You decide at 80% you are fairly good but happy to wait 5 seconds for completion / 500ms for time to first token. If either breaches you give the current answer.
But if you are at 5% you wait for 60s total/2s for a token since the upside of that unspoken model is much higher.
Basically wagering time for quality in a dynamic prediction market in front of the LLM.
I.e. you get 3 replies. 80% confidence. You decide at 80% you are fairly good but happy to wait 5 seconds for completion / 500ms for time to first token. If either breaches you give the current answer.
But if you are at 5% you wait for 60s total/2s for a token since the upside of that unspoken model is much higher.
Basically wagering time for quality in a dynamic prediction market in front of the LLM.