Comment by godelski

  > the model can remember stuff as long as it’s in the context.

You would need an infinite context or compression

Also you might be interested in this theorem

https://en.wikipedia.org/wiki/Data_processing_inequality