While this is a well written paper, I'm not sure it's really contextualizing realistic risks that may arise from AI.
It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit.
It feels like they always anthropomorphize AI as some sort of "God".
The "AI Powered States" aspect is definetly pure sci-fi. Technocratic states have been attempted, and econometrics literally the exact same mathematical models used in AI/ML (Shapely values are an Econometrics tool, Optimization Theory itself got it's start thanks to GosPlan and other attempts and modeling and forecasting economic activity, etc).
As we've seen with the Zizian cult, very smart people can fall into a fallacy trap of treating AI as some omnipotent being that needs to either be destroyed or catered to.
It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit.
It feels like they always anthropomorphize AI as some sort of "God".
The "AI Powered States" aspect is definetly pure sci-fi. Technocratic states have been attempted, and econometrics literally the exact same mathematical models used in AI/ML (Shapely values are an Econometrics tool, Optimization Theory itself got it's start thanks to GosPlan and other attempts and modeling and forecasting economic activity, etc).
As we've seen with the Zizian cult, very smart people can fall into a fallacy trap of treating AI as some omnipotent being that needs to either be destroyed or catered to.