This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior?
I started looking and it turns out that OpenAI is keenly aware of the hallucination problem, and even wrote about how they're correcting for it in this blog post about InstructGPT: https://openai.com/blog/instruction-following/
To summarize: it seems OpenAI is addressing this by taking human-labeled output data from GPT and feeding this into new models, using a reward function to train the future models to behave with less hallucination. This way of working struck me as a mix of software engineering and crop breeding.
They discuss the trade-offs involved with doing this. The "hallucination" effect is probably one of the features that make GPT so creative.
I started looking and it turns out that OpenAI is keenly aware of the hallucination problem, and even wrote about how they're correcting for it in this blog post about InstructGPT: https://openai.com/blog/instruction-following/
To summarize: it seems OpenAI is addressing this by taking human-labeled output data from GPT and feeding this into new models, using a reward function to train the future models to behave with less hallucination. This way of working struck me as a mix of software engineering and crop breeding.
They discuss the trade-offs involved with doing this. The "hallucination" effect is probably one of the features that make GPT so creative.