This high dimensional vector space does allows them to theoretically embed a interpretable "deeper understanding", reference https://www.youtube.com/watch?v=wjZofJX0v4M&t=624s. It's the pre-train process that determines whether or not they demonstrate this "deeper understanding".
This "deeper understanding" refers to the understanding to the word, can be contextual, pragmatic, or semantic.
This "deeper understanding" refers to the understanding to the word, can be contextual, pragmatic, or semantic.