Comment by quatrefoil

While a lot of attention has been given to books3, another large component of this dataset is the deceptively-named "OpenWebText2". What's that? It's a scrape of 15 years' worth of third-party websites that were linked to from upvoted Reddit submissions. I know this includes some of my writing.