Great post. I'm a first-year CS student without much math knowledge (also poor English). I'd like to check whether I got the key idea right.
There are layers and each layer stores key, value pairs (normalized in this article).
If I query Q then Q will be compared with every stored key "Q@A + Q@B + Q@C + ... = value".
If A is the answer then Q@A will be 1, others will be interferences.
D dimension only has d orthogonal directions, so if I store more keys than the dimension, I can't avoid remaining interferences.
What we want is reducing them by going through function F.
And this article introduces heuristic capacity results with different memory store functions (linear -> ReLU -> self attention).
Is it roughly right?
self attention is the best way to suppress interferences?
self attention is the best way to suppress interferences?