Comment by mapontosevenths

The author describes the model's pairing operation as:

(zt,zt+Delta) where Delta is a positive offset.

This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.

I don't know if that's just the paper, or if the code was also b0rk3d.