Lots of ways to make it fail. Not to be rude but you're late to the party. Transit questions is my favorite. Ask it what stations lines 1 and 2 have in common (city of your choice). Nearly 100% of the time there's at least one wrong answer on the list. Ask it what trains go to that station, it likely wont list lines 1 and 2. Point out the contradiction, it will make a new list with new mistakes.
Another good one. Ask for random numbers. They usually aren't much random at all. Ask it what distribution it picked from, it will say it used pythons rng. You and I both know it can't invoke the python interpreter. It can't honestly tell you the non random pattern named randomness it has learned to recall.
Anything to do with nested narrative scope. Ask it to make a story about a conversation with another user who prompted X. It will often conflate the role of each person in the story. It once gave me a narrative about a user who was uncomfortable with a prompt chatGPT had given to it...funny role reversal there. Any attempt to make it produce a transcript of conversation which itself contains some transcript of another conversation as an object of discussion goes over its head. It can't nest.
Another good one. Ask for random numbers. They usually aren't much random at all. Ask it what distribution it picked from, it will say it used pythons rng. You and I both know it can't invoke the python interpreter. It can't honestly tell you the non random pattern named randomness it has learned to recall.
Anything to do with nested narrative scope. Ask it to make a story about a conversation with another user who prompted X. It will often conflate the role of each person in the story. It once gave me a narrative about a user who was uncomfortable with a prompt chatGPT had given to it...funny role reversal there. Any attempt to make it produce a transcript of conversation which itself contains some transcript of another conversation as an object of discussion goes over its head. It can't nest.
That's all I got for now.