Comment by nephanth

Btw, why do transformers have a limit input size in the first place? I'm pretty sure the self-attention mechanisms scale (although with bad complexity) to arbitrary sizes