It's only strange if you think it's just word salad[1].
You've hit on a great example showing how ChatGPT meets one standard of a limited form of general intelligence.
It makes perfect sense if you're not denying that.
But how to explain this while denying it?
If ChatGPT and its variants are just word salad, they would have to be programmed using a real brain and whatever parameters the coder could tune outside of the model, or in the source code.
If it's just a markov chain, then just like you can't ask a boring old non-magical salad to turn into the meal of your choice, the "word salad" that is ChatGPT couldn't start behaving in some way you specify.
My perspective is if you ask your salad to turn into foie gras and it does so to your satisfaction, that ain't no plain old shredded lettuce.
[1] https://en.wikipedia.org/wiki/Word_salad
You've hit on a great example showing how ChatGPT meets one standard of a limited form of general intelligence.
It makes perfect sense if you're not denying that.
But how to explain this while denying it?
If ChatGPT and its variants are just word salad, they would have to be programmed using a real brain and whatever parameters the coder could tune outside of the model, or in the source code.
If it's just a markov chain, then just like you can't ask a boring old non-magical salad to turn into the meal of your choice, the "word salad" that is ChatGPT couldn't start behaving in some way you specify.
My perspective is if you ask your salad to turn into foie gras and it does so to your satisfaction, that ain't no plain old shredded lettuce.
[1] https://en.wikipedia.org/wiki/Word_salad