> LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens.
The mechanisms which are built during training in the big blob of bits we call weights are anything but transparent. How they predict the next word is the big thing here. Saying they ‘just’ predict the next word is ignoring basically everything that actually matters.
The mechanisms which are built during training in the big blob of bits we call weights are anything but transparent. How they predict the next word is the big thing here. Saying they ‘just’ predict the next word is ignoring basically everything that actually matters.