Comment by chemicalnovae

I think _small_ is still doing a lot of work here. Yes, an MoE model doesn't "use" all of it's weights at once _but_ what it does use is still a hell of a lot heavier than the 20K toy shown in the paper.