> The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.