> in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE