Comment Comment by tweedler290 This is really cool!Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k? parent tweedler290 1 month ago
Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?