Comment by aheilbut

is it possible to distill a large model into a (even) smaller MoE model, like OLMoE?