Comment by pizza

You’re correct. There are lots of ways to project a string of embeddings back onto a single embedding - lots of times however these simpler methods do something like pooling the sum if the word embeddings by averaging and hoping that the answer embedding has a small angle with that projection (cosine similarity). You could try adding a way to focus on individual tokens, pairwise interactions, and more sophisticated ways of pooling and projecting embeddings, and then boom you’ve basically just created transformers.