You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.