I don't have access to ChatGPT4, but in my tests I could observe that it can't do some very simple tasks:
- It can't play tic-tac-toe,
- It can't play hangman,
- It insists that winning on stone-paper-scissor using the chat (playing before me) is a matter of probability.
It was also demonstrated that it can't reverse strings.
Actually a transformer doesn't accesses 'strings', all it processes are tokens which are then mapped to vectors by whatever embedding is applied. I think it will be extremely difficult for a transformer to do any of these tasks correctly until a successor model is adopted.
I don't have much hope of any reasonably complex symbolic processing of anything that it was not trained on. Some of these tasks are easy for a human to perform with paper and pencil and a set of rules; of course a human may get confused, but for that you write programs. Write code is one of GPT's skills but It is not "that" good with code for problems that are not mere small modification of problems it was trained on.
EDIT: Could have expressed myself better: I don't have access to chatGPT4; I tested using the "available" chatGPT, I think it is 3.5.
A transcript of me trying to play tic-tac-toe with it: https://pastebin.com/V1CW5hpt
Actually a transformer doesn't accesses 'strings', all it processes are tokens which are then mapped to vectors by whatever embedding is applied. I think it will be extremely difficult for a transformer to do any of these tasks correctly until a successor model is adopted.
I don't have much hope of any reasonably complex symbolic processing of anything that it was not trained on. Some of these tasks are easy for a human to perform with paper and pencil and a set of rules; of course a human may get confused, but for that you write programs. Write code is one of GPT's skills but It is not "that" good with code for problems that are not mere small modification of problems it was trained on.
EDIT: Could have expressed myself better: I don't have access to chatGPT4; I tested using the "available" chatGPT, I think it is 3.5.
A transcript of me trying to play tic-tac-toe with it: https://pastebin.com/V1CW5hpt