At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
It feels like that would be the expected result, all else equal.