I find all the writeups of self-attention very confusing. They seem to inevitably explain small bits in enormous detail and then leave all kinds of giant areas unexplained, usually are replete with jargon that they don't bother to clarify, etc. I feel like most of these are written by people who are confused themselves, trying to clarify the parts they don't understand as if this is what everybody else must be confused about to, so they inevitably gloss over everything they already know.
Fundamentally, I have never understood why a conventional neural network cannot learn self-attention. If layers are fully connected then they definitely have the ability to create a weighting of the inputs accommodating learning of relative spatial features in the data. In fact that's almost a definition of what a neural network is. If relative positional learning is important, that could be added the same way it is in Transformers without the explicit self-attention layer. So what is self-attention really doing beyond this? Why do we need a Query-Key-Value construct here? I am sure I am missing something very basic and fundamental here.
Fundamentally, I have never understood why a conventional neural network cannot learn self-attention. If layers are fully connected then they definitely have the ability to create a weighting of the inputs accommodating learning of relative spatial features in the data. In fact that's almost a definition of what a neural network is. If relative positional learning is important, that could be added the same way it is in Transformers without the explicit self-attention layer. So what is self-attention really doing beyond this? Why do we need a Query-Key-Value construct here? I am sure I am missing something very basic and fundamental here.