I'm quite skeptical of analyses like this one, because I doubt the metrics themselves. Emergence is something that is intuitively noticed by human observers. The desire to quantify everything then leads to the creation of (imperfect) metrics designed to capture what the observers already know. Those same metrics are then taken as the definition of the properties said to be emergent, and articles like this one are among the consequences of that choice.
The paper's claim is essentially "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities, so emergence isn't a real phenomenon".
But an equally valid interpretation would be "none of these metrics actually capture the properties we are truly interested in". Which, given the complexity of what we are dealing with here, seems entirely reasonable. It's not like we suddenly learned how to accurately quantify performance at language tasks. The whole reason LLMs are so great in the first place is because traditional 'mechanical' language models suck so bad.
The paper's claim is essentially "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities, so emergence isn't a real phenomenon".
But an equally valid interpretation would be "none of these metrics actually capture the properties we are truly interested in". Which, given the complexity of what we are dealing with here, seems entirely reasonable. It's not like we suddenly learned how to accurately quantify performance at language tasks. The whole reason LLMs are so great in the first place is because traditional 'mechanical' language models suck so bad.