A lot of previous transformer-based models did not have great depth simply because "shallow" transformers already took forever to train.
I am concerned that we are approaching an age where only massive companies and research groups with tons of GPU resources will be able to train next-gen models.
I am concerned that we are approaching an age where only massive companies and research groups with tons of GPU resources will be able to train next-gen models.