This project is now all the more interesting given what we now know about LLM scaling laws. It will take trillions of high quality, factual tokens to train the next generation of LLMs.
Copyright is also appearing to be beside the point. A petabyte of copyrighted data could be run through an LLM and rephrased/summarized in a way that retains the essential knowledge but creates a public domain dataset for further training.
Copyright is also appearing to be beside the point. A petabyte of copyrighted data could be run through an LLM and rephrased/summarized in a way that retains the essential knowledge but creates a public domain dataset for further training.