I don't understand why they're comparing the parameter sizes to lines of code.
AFAIK you can just increase the layer parameters of a 1B model to whatever you want? Like, the difference between a 1B and 175B model can be just changing a few numbers, and not adding any LOC at all?
LOC has never been a limitation for large models, it's been the compute+training data required.
Most of the LOC is spent on optimization, and they don't address MoE or anything fancy like that?
AFAIK you can just increase the layer parameters of a 1B model to whatever you want? Like, the difference between a 1B and 175B model can be just changing a few numbers, and not adding any LOC at all?
LOC has never been a limitation for large models, it's been the compute+training data required.
Most of the LOC is spent on optimization, and they don't address MoE or anything fancy like that?