> a world where AI is indistinguishable, or far superior, to human intelligence
I think the part about being "indistinguishable from human intelligence" is potentially a intellectual trap. We might get to it being far superior while still underperforming at some tasks or behaving in ways that don't make sense to a human mind. An AI mind will highly likely work completely differently from humans and communicating with it should be more thought of as communicating with a quite foreign alien than with a human trapped in a computer.
As a comparison, I'm sure there are some tasks in which some animals do better than humans. Yet no human would conclude that humans are inferior to some monkey who might find its way around the rain forest better or whatever we are worse at.
I think the part about being "indistinguishable from human intelligence" is potentially a intellectual trap. We might get to it being far superior while still underperforming at some tasks or behaving in ways that don't make sense to a human mind. An AI mind will highly likely work completely differently from humans and communicating with it should be more thought of as communicating with a quite foreign alien than with a human trapped in a computer.
As a comparison, I'm sure there are some tasks in which some animals do better than humans. Yet no human would conclude that humans are inferior to some monkey who might find its way around the rain forest better or whatever we are worse at.