Comment by mike_hearn

Aren't the models already doing this, in a way? We know they can do things like write rhyming poems and song lyrics that do make perfect sense, so at some point the activations must be encoding some sort of overall plan for the upcoming sentences, even if maybe every word isn't predicted yet.