Comment by read_if_gay_

however, if you need to swap experts on each token, you might as well run on cpu.