Comment by bigfishrunning

But those scores are only the likelihood of the given token being the next one in a plausible phrase, not that the underlying data is correct.