Comment by guy98238710

There's one aspect I never saw explained. Why is masking used instead of a sliding window? Why even bother with masking when future tokens can be easily hidden by simply positioning the context window before the current token? Isn't sliding window optimal for maximizing context available to the model? Is masking done because moving the window would impact computational cost or output stability/quality? Can anyone shed light on this?