Comment by karmasimida

But you do need to hold all 128 experts in memory? Or not?

Or they simply consider inference efficiency as latency