The "fast takeoff" scenarios got most of the attention, but the "slow takeoff" scenarios better match the reality we find ourselves in and they tend to involve economics.
We are a capitalist economy, which means the control plane is privately owned, which gives a superintelligent entity paths to power and resource accumulation that are not only safeguarded from public interference by design but also have a built-in time-tested mechanism for aligning insiders and outsiders to their goals, participating in politics, and so on.
Instead of "1. AI builds human killer virus, 2. ..., 3. profit," this apocalypse looks like a company that grows by providing genuine goods and services, massive returns for its boosters and allies in politics, the ASI uses that wealth to build independent infrastructure for itself, and then it gets rid of the stinky smelly humans with a virus so that it can turn the iron in our blood into paperclips, or whatever.
One can hope that GPUs will never be able to run a company well enough to acquire bootstrap-level resources or that "you are a helpful assistant" will keep them aligned to someone, but frankly I suspect those are both cope and the real hope lies in the first attempts failing in spectacular enough fashion that people decide to do something to prevent it going forward. Or in "friendly AIs" saturating the competitive landscape first -- that's a bit too Disney for my cynical taste, but I would much prefer it to the alternative.
We are a capitalist economy, which means the control plane is privately owned, which gives a superintelligent entity paths to power and resource accumulation that are not only safeguarded from public interference by design but also have a built-in time-tested mechanism for aligning insiders and outsiders to their goals, participating in politics, and so on.
Instead of "1. AI builds human killer virus, 2. ..., 3. profit," this apocalypse looks like a company that grows by providing genuine goods and services, massive returns for its boosters and allies in politics, the ASI uses that wealth to build independent infrastructure for itself, and then it gets rid of the stinky smelly humans with a virus so that it can turn the iron in our blood into paperclips, or whatever.
One can hope that GPUs will never be able to run a company well enough to acquire bootstrap-level resources or that "you are a helpful assistant" will keep them aligned to someone, but frankly I suspect those are both cope and the real hope lies in the first attempts failing in spectacular enough fashion that people decide to do something to prevent it going forward. Or in "friendly AIs" saturating the competitive landscape first -- that's a bit too Disney for my cynical taste, but I would much prefer it to the alternative.