Comment by Sivart13

"Confidently wrong" is definitely a standard behavior model for LLMs.

I agree with the other reply that you're likely to get better results if it has some kind of test case to run that's more authoritative than its own reasoning.