It's not very good at giving the proper credence to version numbers.
Granted I started with a hard one, but I asked it how to create a GTK3 interface with PHP, and it gave me instructions to download and use an abandoned project for GTK2, but described it as GTK3 in the steps.
I tried asking it some other questions about languages and applications specific to version numbers - it seems to provide incredibly ambiguous and version agnostic responses, or tells me essentially "you may or may not be able to do this, and you should check if you can" when the answer is clearly that it is not possible. Or it just ignores the version entirely and provides instructions that don't match up - hallucinating UI elements or commands that don't (or didn't yet) exist.
For something targeted at developers, this is a gaping hole and is what I would consider a major oversight - the responses I'm getting are very similar in content to what I get from GPT and Ollama's generic models.
Granted I started with a hard one, but I asked it how to create a GTK3 interface with PHP, and it gave me instructions to download and use an abandoned project for GTK2, but described it as GTK3 in the steps.
I tried asking it some other questions about languages and applications specific to version numbers - it seems to provide incredibly ambiguous and version agnostic responses, or tells me essentially "you may or may not be able to do this, and you should check if you can" when the answer is clearly that it is not possible. Or it just ignores the version entirely and provides instructions that don't match up - hallucinating UI elements or commands that don't (or didn't yet) exist.
For something targeted at developers, this is a gaping hole and is what I would consider a major oversight - the responses I'm getting are very similar in content to what I get from GPT and Ollama's generic models.