Yeah, that's seems to be my impression as well. Of course it's true that in a sense all it does is capturing statistical relationships - because that's all that is there in the input data. However what kind of relationships it captures may still be arbitrarily complex (within the contraints of the model architecture).
We know it's more complex than just "empirical probability of word x given the n words before", because that would result in a markov chain and we know those don't generate the kind of output we are seeing.
We also see that it's able to map descriptions of tasks to its execution, even for unseen tasks. E.g., I can tell it "Write a limerick that contains the names of all living US presidents and format is as a JSON array inside a python script.". The result will probably contain some dead US presidents or some canadian prime ministers or whatever, and the text may be a haiku and not a limerick - but the output will usually be a python function with a JSON array with a poem with some names in it.
I don't see how that could work without a more abstract representation of the concepts "president's names", "poem", "json" and "python", so it can combine them meaningfully into a single response.
We know it's more complex than just "empirical probability of word x given the n words before", because that would result in a markov chain and we know those don't generate the kind of output we are seeing.
We also see that it's able to map descriptions of tasks to its execution, even for unseen tasks. E.g., I can tell it "Write a limerick that contains the names of all living US presidents and format is as a JSON array inside a python script.". The result will probably contain some dead US presidents or some canadian prime ministers or whatever, and the text may be a haiku and not a limerick - but the output will usually be a python function with a JSON array with a poem with some names in it.
I don't see how that could work without a more abstract representation of the concepts "president's names", "poem", "json" and "python", so it can combine them meaningfully into a single response.