Here's one piece of evidence suggesting it's more like rote pattern matching than reasoning.
> All the signs in this building are written in mirror writing. A glass door has ‘push’ written on it in mirror writing. Should you push or pull it
>> If the sign on the glass door is written in mirror writing and says "push," then you should actually pull the door. This is because the mirror writing makes the text appear reversed, so the word "push" would appear as "hsup" in a mirror, which could cause confusion for someone trying to enter the building. Therefore, pulling the door would be the correct action to take.
(Latest chat.openai.com, so if I'm reading the promo materials right that's gpt4)
> All the signs in this building are written in mirror writing. A glass door has ‘push’ written on it in mirror writing. Should you push or pull it
>> If the sign on the glass door is written in mirror writing and says "push," then you should actually pull the door. This is because the mirror writing makes the text appear reversed, so the word "push" would appear as "hsup" in a mirror, which could cause confusion for someone trying to enter the building. Therefore, pulling the door would be the correct action to take.
(Latest chat.openai.com, so if I'm reading the promo materials right that's gpt4)