Very good point. This is an active area of research, there's been some work in using other similarity measures like MI,Kendall's Tau etc in embeddings as well. And they have been found to perform pretty well.
EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these
PS: I am the author and just wanted to talk about the familiar basic stuff on this one
I found something for MI if you wanna check it out : https://www.aclweb.org/anthology/2020.acl-main.741.pdf
EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these
PS: I am the author and just wanted to talk about the familiar basic stuff on this one