> Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do.
It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.
It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.
i.e. it would look immoral to us.
(There's also the thing about civilization already being aligned against the ecosystem and against the happiness of its constituents, but that's kind of a separate discussion.)
In other words, what is good for our sanity or our lakes may be bad for GDP. (Well I guess you don't need superintelligence to tell you that.) We already know what we need to do and we're already not willing to do it.
It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.
It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.
i.e. it would look immoral to us.
(There's also the thing about civilization already being aligned against the ecosystem and against the happiness of its constituents, but that's kind of a separate discussion.)
In other words, what is good for our sanity or our lakes may be bad for GDP. (Well I guess you don't need superintelligence to tell you that.) We already know what we need to do and we're already not willing to do it.