Comment by wenhan_zhou

Yep. Or even better, compact after a random number of turns. The model must then learn to preserve useful context at arbitrary context lengths.