This is so well written, deep, funny, and sad.
> Looking up at the stars in wonderment, or enjoying a complex puzzle game, or being inspired by an encouraging teacher, and dreaming that one day an unfeeling corporate robot[1] will burn a rainforest to compile a list of 700 highly obscure barely-readable facts that might be true that nobody can understand as part of a marketing gimmick to raise hype for an IPO[2].
But if I am going to critique it's main point it would be this part:
> in the pursuit of advancing humanity’s knowledge as quickly as possible, many academics will recoil in anger at the suggestion that it is worth spending any time making results understandable to humans. It is inefficient to explain or check a result when those resources can be spent generating another.
While obviously ridiculous, I think this does not go far enough. The problem isn't that people don't care and resources should not be spent to explain AI results to human minds, the problem is that AI will soon be better at explaining itself than humans. The puzzle then changes from can AI formally verify and explain a proof to: do human minds trust the proof. Do we believe that the proof is correct and not an elaborate AI hallucination using tricks that are beyond our understanding. In order to trust it a formal verification is not enough unless it is human understandable. In fact human understandability is the only thing that matters. That is something that AI can never do on its own.
> Looking up at the stars in wonderment, or enjoying a complex puzzle game, or being inspired by an encouraging teacher, and dreaming that one day an unfeeling corporate robot[1] will burn a rainforest to compile a list of 700 highly obscure barely-readable facts that might be true that nobody can understand as part of a marketing gimmick to raise hype for an IPO[2].
But if I am going to critique it's main point it would be this part:
> in the pursuit of advancing humanity’s knowledge as quickly as possible, many academics will recoil in anger at the suggestion that it is worth spending any time making results understandable to humans. It is inefficient to explain or check a result when those resources can be spent generating another.
While obviously ridiculous, I think this does not go far enough. The problem isn't that people don't care and resources should not be spent to explain AI results to human minds, the problem is that AI will soon be better at explaining itself than humans. The puzzle then changes from can AI formally verify and explain a proof to: do human minds trust the proof. Do we believe that the proof is correct and not an elaborate AI hallucination using tricks that are beyond our understanding. In order to trust it a formal verification is not enough unless it is human understandable. In fact human understandability is the only thing that matters. That is something that AI can never do on its own.