And hesitated here. It seems then the article is predicated on us trying to store "white noise". However in the real world, data has much more structure than that. Every bit of structure is an opportunity to compress more. So all the conclusions here would be an upper/lower bound rather than a true estimate.
I assume this is accounted for by:
> In fact we haven’t actually considered the training data at all, which affects what actually is being represented in the keys and values
but it understates the point: much of the compressability as well as intelligence in the models come explicitly from the structure in the data. So it's very interesting you can derive MLP and transformer architecture even absent that. But the real leverage comes from the interaction of those things WITH structure in the data.
I got to this part:
> If the keys are random unit vectors
And hesitated here. It seems then the article is predicated on us trying to store "white noise". However in the real world, data has much more structure than that. Every bit of structure is an opportunity to compress more. So all the conclusions here would be an upper/lower bound rather than a true estimate.
I assume this is accounted for by:
> In fact we haven’t actually considered the training data at all, which affects what actually is being represented in the keys and values
but it understates the point: much of the compressability as well as intelligence in the models come explicitly from the structure in the data. So it's very interesting you can derive MLP and transformer architecture even absent that. But the real leverage comes from the interaction of those things WITH structure in the data.