That's not how I read it.
For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.
For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).
For a given t, this produces up to m pairs:
(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….
We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.
So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.