Comment by itkovian_

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.