I saw someone describe LLM hallucinations and inaccuracies as "compression artifacts." This thing has seen all the data that can possibly be fed to it and stored it in a much smaller size, so it's not crazy to think there will be compression artifacts. I have noticed that the quality of its answers seems to go down significantly when I am asking about a topic about which there isn't much information on google.
But it makes you wonder if future LLMs aren't going to suffer from the JPEG of a JPEG of a JPEG effect where it will be impossible to train an LLM off of new data that isn't generated by another LLM.
But it makes you wonder if future LLMs aren't going to suffer from the JPEG of a JPEG of a JPEG effect where it will be impossible to train an LLM off of new data that isn't generated by another LLM.