I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models
It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still possible
Maybe at some point someone figures out how to train models with a backspace token
It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still possible
Maybe at some point someone figures out how to train models with a backspace token