In this context, what does "closest" mean? Most similar token embeddings? Most similar Q embedding to prior K embeddings? Most similar positional embedding?
Are you, in essence, implying that the Key-Query attention mechanism is basically a neural function approximation of the Nadaraya-Watson kernel or something very like it?
Are you, in essence, implying that the Key-Query attention mechanism is basically a neural function approximation of the Nadaraya-Watson kernel or something very like it?