> What will replace Internet data for training? Curated synthetic datasets?
Perhaps a different take at this could be: if I wanted to train "state law" LLM that is exceedingly good in interpreting state law, what are the obstacles to download all the law and regulations material for given state that will allow me to train LLM such that it becomes 95th percentile of all law trainees and lawyers.
In that case, and my point is, that we already don't need an "Internet". We just need a sufficiently sized and curated domain-specific dataset and the result we can get is already scary. "State law" LLM was just an example but the same logic applies to basically any other domain - want a domain-specific (LLM) expert? Train it.
Perhaps a different take at this could be: if I wanted to train "state law" LLM that is exceedingly good in interpreting state law, what are the obstacles to download all the law and regulations material for given state that will allow me to train LLM such that it becomes 95th percentile of all law trainees and lawyers.
In that case, and my point is, that we already don't need an "Internet". We just need a sufficiently sized and curated domain-specific dataset and the result we can get is already scary. "State law" LLM was just an example but the same logic applies to basically any other domain - want a domain-specific (LLM) expert? Train it.