I only recently started learning about vector embeddings, and the thing I can't wrap my head around is why cosine similarity should be so effective when (to my understanding) it should only capture linear similarity.
Shouldn't that "linear" constraint have more of a blunting effect on embeddings effectiveness? Why not use a different metric like mutual information that can account for non-linearities. Whenever correlation comes up in context of statistical comparison, there's always this warning that correlation is limited as a way of comparing data due to it's linearity so I'm confused why it seems so effective in the context of embeddings.
Shouldn't that "linear" constraint have more of a blunting effect on embeddings effectiveness? Why not use a different metric like mutual information that can account for non-linearities. Whenever correlation comes up in context of statistical comparison, there's always this warning that correlation is limited as a way of comparing data due to it's linearity so I'm confused why it seems so effective in the context of embeddings.