> So, these models are basically within 1% of each other in terms of final pretraining loss.
How is this loss calculated though? Since it is called "loss" and not "performance metric", I'm going to assume it is the teacher forced cross entropy loss.
I'm not too familiar with LLM training but having been doing some fair amount of seq2seq training lately in other domains I've observed that the relationship between "loss" and autoregressive inference performance gets very narrow towards the end of training. What I mean is that smaller and smaller reductions in loss lead to better improvements in the autoregressive output. So at least I have some doubt that in practice that 1% loss improvement is not actually incredibly significant with respect to how well the model actually performs at inference time.
But I'm pretty interested in this topic and if people here have observed this or the contrary I'd be curious to know.
How is this loss calculated though? Since it is called "loss" and not "performance metric", I'm going to assume it is the teacher forced cross entropy loss.
I'm not too familiar with LLM training but having been doing some fair amount of seq2seq training lately in other domains I've observed that the relationship between "loss" and autoregressive inference performance gets very narrow towards the end of training. What I mean is that smaller and smaller reductions in loss lead to better improvements in the autoregressive output. So at least I have some doubt that in practice that 1% loss improvement is not actually incredibly significant with respect to how well the model actually performs at inference time.
But I'm pretty interested in this topic and if people here have observed this or the contrary I'd be curious to know.