Comment by nfiedel

OpenAI trained a large (1.5B parameter) Transformer model called GPT-2 on a diverse set of pages from the web. From their paper, GPT-2 "achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting"

Blog entry with link to the paper: https://openai.com/blog/better-language-models/