> Yoshua Bengio has written about a scenario where an AI is asked to solve climate change, and it designs a virus that decimates humans. Presto. No more humans, no more climate change.
> This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.
Paraphrasing:
> Yoshua Bengio has written about a scenario where an AI is asked to solve climate change, and it designs a virus that decimates humans. Presto. No more humans, no more climate change.
> This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.