Comment by ieatpies

Attention based models are still extremely data inefficient and fail to generalize. I recommend reading the "On the Measure of Intelligence" paper by Chollet if this topic interests you.