This feels like giving up and accepting that LLMs are just magic - or "emergence" to use a modern term which is practically used in the same sense.
I think there are a few practical questions which we can use to gauge the level of understanding we have: Do we know which parts of the architecture and the training process are actually essential and which can be left away? Do we know which of the weights are essential? Do we know how the network arrives at a particular token probability which suggests some deep, abstract understanding of the prompt? Or likewise, if the network arrives at an incorrect answers, can we say which exact part of the calculation went wrong?
Or for the current thread, can we explain how the network decides when to treat a text as an instruction and when as data? (Because it certainly does treat parts of the text as data: I can prompt it to translate a sentence into a different language and this will also often work with imperative sentences, but not always - if the imperative sentence is formulated in the right way, the network will treat it as an instruction.)
I think there are a few practical questions which we can use to gauge the level of understanding we have: Do we know which parts of the architecture and the training process are actually essential and which can be left away? Do we know which of the weights are essential? Do we know how the network arrives at a particular token probability which suggests some deep, abstract understanding of the prompt? Or likewise, if the network arrives at an incorrect answers, can we say which exact part of the calculation went wrong?
Or for the current thread, can we explain how the network decides when to treat a text as an instruction and when as data? (Because it certainly does treat parts of the text as data: I can prompt it to translate a sentence into a different language and this will also often work with imperative sentences, but not always - if the imperative sentence is formulated in the right way, the network will treat it as an instruction.)