Comment by aiddun

Speculative decoding does this to an extent - using a smaller model to generate its own predictions and putting them in the batch of the bigger model until they diverge

https://huggingface.co/blog/whisper-speculative-decoding