Comment by dmrivers

Attention replaced recurrence over tokens in 2017, this does the same over depth of the layers. It's apparently not an entirely new idea, but also an elegant reapplication of the attention mechanism.