Comment by rar00

This argument works better for state space models. A transformer would still steps context one token at a time, not maintain an internal 1e18 state.