I mostly agree, but struggle with saying this with perfect certainty.
Understanding in the "have mental model of the world, apply it, derive thoughts from that model, derive words from thoughts" pattern is a thing they don't do the way we do.
But understanding of some kinds CAN be encoded into tokens and their relationships. They're clearly capable of novel, correct inferences, that are not directly contained within their training sets.
I all-but-guarantee my "My fish suffocated when I brought it to space, even though I gave it a space suit filled with pure water, why?" test case is not something it was explicitly trained on, but it correctly inferred "Because fish need oxygenated water"
Understanding in the "have mental model of the world, apply it, derive thoughts from that model, derive words from thoughts" pattern is a thing they don't do the way we do.
But understanding of some kinds CAN be encoded into tokens and their relationships. They're clearly capable of novel, correct inferences, that are not directly contained within their training sets.
I all-but-guarantee my "My fish suffocated when I brought it to space, even though I gave it a space suit filled with pure water, why?" test case is not something it was explicitly trained on, but it correctly inferred "Because fish need oxygenated water"