How does one select a good candidate for the draft model in speculative decoding? I imagine that there's some better intuition than just selecting the next parameter count down (i.e 70B -> 13B, 13B -> 7B).
Also how does that interact with MoE models? Do you have a mini version of the MoE, with smaller experts?
Also how does that interact with MoE models? Do you have a mini version of the MoE, with smaller experts?