The thesis and opening section is interesting for sure, because it shows how mechanisms and incentives that exist today could lead to really bad outcomes. However then it leaps to:
> Because this disempowerment would be global and permanent, and because human flourishing requires substantial resources in global terms, it could plausibly lead to human extinction or similar outcomes.
Which seems a bit over the top. AI is way up at the top of a huge pyramid of infrastructure and energy dependence. It's nowhere near self-sustaining without a huge amount of human input for the foreseeable future. Therefore the disempowerment isn't permanent or absolute, it's just another in a long line of technological tools that enable consolidation of power by reducing the need for human labor and enabling new, previously uneconomical approaches to different human goals. If at any point the disempowerment reaches a tipping point affecting the masses and those in control aren't paying attention then I expect standard revolution dynamics would take over.
> Because this disempowerment would be global and permanent, and because human flourishing requires substantial resources in global terms, it could plausibly lead to human extinction or similar outcomes.
Which seems a bit over the top. AI is way up at the top of a huge pyramid of infrastructure and energy dependence. It's nowhere near self-sustaining without a huge amount of human input for the foreseeable future. Therefore the disempowerment isn't permanent or absolute, it's just another in a long line of technological tools that enable consolidation of power by reducing the need for human labor and enabling new, previously uneconomical approaches to different human goals. If at any point the disempowerment reaches a tipping point affecting the masses and those in control aren't paying attention then I expect standard revolution dynamics would take over.