This is an interesting perspective and maybe the most interesting argument in the thread.
But why can't a hyper-intelegent AI also do this biology-esque math work of explaining its work to feeble humans?
It is easy for a genius to have theory of mind over an idiot but not vice-versa. So it should be easier for a genius AI to explain its work to an idiot human than for an idiot human to dissect the work of an genius AI.
Also, we're not talking about solutions that are arrived though an evolutionary/statistical process like explaining the design of an organism or the weights of an LLM. We are talking about solutions that are arrived through lingual/textual/rational processes like mathematics. So it's fundamentally a different mode of thinking.
It's not that we have "a brain to study", it's like if we had an "explaination of how a brain works" to study.
The task of breaking down these AI proofs is not translating and analyzing a fuzzy real-world system into a hard logical system (like in biology), but in translating a hard logical system into a simpler hard-logical system we can understand.
But why can't a hyper-intelegent AI also do this biology-esque math work of explaining its work to feeble humans?
It is easy for a genius to have theory of mind over an idiot but not vice-versa. So it should be easier for a genius AI to explain its work to an idiot human than for an idiot human to dissect the work of an genius AI.
Also, we're not talking about solutions that are arrived though an evolutionary/statistical process like explaining the design of an organism or the weights of an LLM. We are talking about solutions that are arrived through lingual/textual/rational processes like mathematics. So it's fundamentally a different mode of thinking.
It's not that we have "a brain to study", it's like if we had an "explaination of how a brain works" to study.
The task of breaking down these AI proofs is not translating and analyzing a fuzzy real-world system into a hard logical system (like in biology), but in translating a hard logical system into a simpler hard-logical system we can understand.