Comment Comment by 1R053 the paper with details: https://arxiv.org/pdf/2411.02265They use- 16 experts, of which one is activated per token- 1 shared expert that is always activein summary that makes around 52B active parameters per token instead of the 405B of LLama3.1. parent 1R053 2 years ago
They use
- 16 experts, of which one is activated per token
- 1 shared expert that is always active
in summary that makes around 52B active parameters per token instead of the 405B of LLama3.1.