This is a common misconception of how MoE models work. To be clear, 17B parameters are activated for each token generated.
In practice you will almost certainly be pulling the full 109B parameters though the CPU/GPU cache hierarchy to generate non-trivial output, or at least a significant fraction of that.
This is a common misconception of how MoE models work. To be clear, 17B parameters are activated for each token generated.
In practice you will almost certainly be pulling the full 109B parameters though the CPU/GPU cache hierarchy to generate non-trivial output, or at least a significant fraction of that.