Comment by andy99

It would be news is somebody showed transformers could generalize beyond the training data. Deep learning models generally cannot, so it's not a surprise this holds for transformers.