It's very simple: I've played for several hours with ChatGPT and yet, despite all the hype, I have not been able to find
one non-trivial task that it can actually accomplish. I want to understand the power and the limitations of LLMs, so I'd be grateful if someone could point me to any example of a non-trivial question that ChatGPT can actually answer.
For the purposes of this question, non-trivial means roughly "well-defined question whose answer is not found in <1min of googling". So no non-well-defined tasks ("write X in the style of Y") and no copy-paste tasks (for example, programming tasks which have a ready-made answer in stackoverflow).
Here is a (non-exhaustive) list of such tasks I've tried, with no success.
- Write a program that solves the TOV equations
- Write a program that decodes a Posit
- Write a program that scores a trick in a simple card game
- Simulate a python terminal (fucked up immediately when I asked to slice a string or to bit-shift an int).
- Tell me the melody of a given Mozart piano sonata
- Prove a basic fact about metric spaces (diameter of open ball is twice the radius)
- Prove a basic fact about quantum gates
- Identify a chord given the pitch classes
- Write a poem with a well-defined rhyme
- Write a poem with a well-defined metric
- Tell me who developed an (obscure) software
- Give me a short biography of an (obscure) singer
- Summarise a research paper
- Translate a joke
- Simulate a Zork-like game
- Summarise the Europa League final 2013
- Summarise the Counterstrike Boston Major final 2018
In every single one of these it failed. Now it's great that it can answer the question "how to I make an http request in js", but that's a google search, not really impressive in my book. As you can see, even some questions that are google searches it failed.
Suggestions and discussion about LLMs welcome!
Also note I only have access to GPT 3.5, not 4.
For the purposes of this question, non-trivial means roughly "well-defined question whose answer is not found in <1min of googling". So no non-well-defined tasks ("write X in the style of Y") and no copy-paste tasks (for example, programming tasks which have a ready-made answer in stackoverflow).
Here is a (non-exhaustive) list of such tasks I've tried, with no success.
- Write a program that solves the TOV equations
- Write a program that decodes a Posit
- Write a program that scores a trick in a simple card game
- Simulate a python terminal (fucked up immediately when I asked to slice a string or to bit-shift an int).
- Tell me the melody of a given Mozart piano sonata
- Prove a basic fact about metric spaces (diameter of open ball is twice the radius)
- Prove a basic fact about quantum gates
- Identify a chord given the pitch classes
- Write a poem with a well-defined rhyme
- Write a poem with a well-defined metric
- Tell me who developed an (obscure) software
- Give me a short biography of an (obscure) singer
- Summarise a research paper
- Translate a joke
- Simulate a Zork-like game
- Summarise the Europa League final 2013
- Summarise the Counterstrike Boston Major final 2018
In every single one of these it failed. Now it's great that it can answer the question "how to I make an http request in js", but that's a google search, not really impressive in my book. As you can see, even some questions that are google searches it failed.
Suggestions and discussion about LLMs welcome!
Also note I only have access to GPT 3.5, not 4.