Am I misunderstanding? The paper says:
> We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations.
But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program?
len, print = print, len
def print_len(x):
"Print the length of x"
len(print(x)) # since print now refers to len function
# Example usage
test_string = "Hello, World!"
print_len(test_string)
I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored.
> We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations.
But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program?
I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored.