Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper)
I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer:
https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64caf06c34...
Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong...
https://chatgpt.com/share/7e6a7201-dd2b-43c6-8427-76e5b003ca...
Also worth noting that there are more complex examples where GPT4o seems to fall down such as:
> Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice's father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice's sister have?
However I can't honestly say that this is THAT simple or that most people would get this right...
I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer:
https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64caf06c34...
Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong...
https://chatgpt.com/share/7e6a7201-dd2b-43c6-8427-76e5b003ca...
Also worth noting that there are more complex examples where GPT4o seems to fall down such as:
> Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice's father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice's sister have?
However I can't honestly say that this is THAT simple or that most people would get this right...