> the problem is that AI will soon be better at explaining itself than humans
(I assume you mean explain itself to humans here.)
I keep seeing people saying this as fact. It might, but it might not, and there are various reasons one might be skeptical that it will, using current techniques. Not least among these is that explaining something well to a human requires a degree of empathy and sophisticated theory-of-mind that LLMs have not yet exhibited yet*, and that is an area where humans are uniquely advantaged by being humans.
Someone will almost certainly reply by just stating as fact that they will achieve this goal, or pointing out that past predictions that LLMs wouldn't be able to do particular things have failed, and that's all very well. I certainly wouldn't be rash enough to claim it's impossible, but there are still models of all this available that are consistent with all the successes so far but don't predict "LLMs can do anything and will do everything". We'll find out in due course.
(I should clarify that the benchmark here is good human explanations. Of course many humans are capable of producing "explanations" worse than Markov chain text, let alone LLM output.)
* My idle speculation is that a lack of decent theory-of-mind is why LLMs still suck at using the "it's not X it's Y" construct. I doubt they actually use it any more frequently than humans when writing on the same topics, but they use it wrong and it's jarring. This particular rhetorical flourish works when there's some reason to think that X is an idea that might actually be present or forming in the mind of someone reading the passage (or in some other mind that they in turn might be imagining, eg. a fictional character), and that explicitly calling it out and negating it will have a particular dramatic effect. LLMs probably use it with about the expected frequency, but they use it for that reason (frequency) and then come up with something to put in the X and Y slots. The result often jars because the X it uses isn't something pertinent to the reader's thought process and/or the Y isn't a convincing, persuasive or relevant counterpoint to it. I suspect the result is that they stick out and seem more frequent than they are.
(I assume you mean explain itself to humans here.)
I keep seeing people saying this as fact. It might, but it might not, and there are various reasons one might be skeptical that it will, using current techniques. Not least among these is that explaining something well to a human requires a degree of empathy and sophisticated theory-of-mind that LLMs have not yet exhibited yet*, and that is an area where humans are uniquely advantaged by being humans.
Someone will almost certainly reply by just stating as fact that they will achieve this goal, or pointing out that past predictions that LLMs wouldn't be able to do particular things have failed, and that's all very well. I certainly wouldn't be rash enough to claim it's impossible, but there are still models of all this available that are consistent with all the successes so far but don't predict "LLMs can do anything and will do everything". We'll find out in due course.
(I should clarify that the benchmark here is good human explanations. Of course many humans are capable of producing "explanations" worse than Markov chain text, let alone LLM output.)
* My idle speculation is that a lack of decent theory-of-mind is why LLMs still suck at using the "it's not X it's Y" construct. I doubt they actually use it any more frequently than humans when writing on the same topics, but they use it wrong and it's jarring. This particular rhetorical flourish works when there's some reason to think that X is an idea that might actually be present or forming in the mind of someone reading the passage (or in some other mind that they in turn might be imagining, eg. a fictional character), and that explicitly calling it out and negating it will have a particular dramatic effect. LLMs probably use it with about the expected frequency, but they use it for that reason (frequency) and then come up with something to put in the X and Y slots. The result often jars because the X it uses isn't something pertinent to the reader's thought process and/or the Y isn't a convincing, persuasive or relevant counterpoint to it. I suspect the result is that they stick out and seem more frequent than they are.