Depends what you mean by "spiral catastrophically".
My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom.
A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact.
"They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously.
Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.
My P(doom in my lifetime) is about 0.1, because I think minor catastrophes will break something hard enough that it doesn't get to escalate to actual-doom.
A recent-ish link was a biologist's blog about why even recent developments wouldn't allow some, I think the example was disgruntled teenager, making a virus that kills everyone. I'm not a biologist, so I'll just take their word as fact.
"They can't kill everyone like that" does not preclude same disgruntled teenager making a virus that kills as many as covid, or the plague, or even just a really bad influenza. If that happens, even "just" 0.1% of the population dying, you can bet your arse the problem will suddenly get taken very seriously.
Only way it spirals is if the potential for doom from each successive AI version goes up faster than the net impact of all the world's idiots and bastards misusing the slightly weaker predecessor. Which isn't impossible, but hence 10%.