The author makes very compelling arguments within the context of known tools/techniques that we have today. However, they conveniently sidestep the possibility of the development of new tools and techniques. This sidestep is overtly stated under the "Active Learning Is Hard & Fundamental" section. And that entire section basically boils down to "I personally believe this is hard and will take a long time."
The only reason we have things like ChatGPT today is because of the surprise development of the transformer. Nobody understands how intelligence fundamentally works, so we literally have no ability to predict when the next groundbreaking architectural development will occur.
And self-improvement concerns are mostly orthogonal to the main risk of AI: the development of AGI. Even a mediocre AGI that is only "decent" at tasks, can be weaponized as a collective superintelligence.
The only reason we have things like ChatGPT today is because of the surprise development of the transformer. Nobody understands how intelligence fundamentally works, so we literally have no ability to predict when the next groundbreaking architectural development will occur.
And self-improvement concerns are mostly orthogonal to the main risk of AI: the development of AGI. Even a mediocre AGI that is only "decent" at tasks, can be weaponized as a collective superintelligence.