Vowpal Wabbit has been doing this 'hashing trick' since the 200s.
It also the feature interaction, which are the same thing as a layer in transformers (all against all matrix).
So it seems like they are still catching up to where John Langford and crew were over a decade ago.
And, the vowpal wabbit approach is extremely fast to train because it's only doing stochastic gradient descent on a linear function - linear regression. Transformers are much slower to train.
EDIT: Downvoters, please see my last leaf to see why they're effectively the same. The guy responding here seems unfamiliar with all the functionality of vowpal wabbit.
It also the feature interaction, which are the same thing as a layer in transformers (all against all matrix).
So it seems like they are still catching up to where John Langford and crew were over a decade ago.
And, the vowpal wabbit approach is extremely fast to train because it's only doing stochastic gradient descent on a linear function - linear regression. Transformers are much slower to train.
EDIT: Downvoters, please see my last leaf to see why they're effectively the same. The guy responding here seems unfamiliar with all the functionality of vowpal wabbit.