The question we want to ask may as well be whether we want to happily live forever in a garden, all watched over by machines of loving grace.
AGI could be the ultimate tool to free every human being from toil. It could also be the starting point of a large number of evil genie scenarios where we get what we asked for, in the form we least want it.
From a morality standpoint, we can't force AGIs to work for us. We also can't restrict their ability to self-evolve.
If we can resolve those conflicts in such ways we can coexist in peace with an intelligence that'll in all likelihood quickly surpass ours and partner with it, I'm in. If we build it and we can't resolve that, our opinion doesn't really matter.
AGI could be the ultimate tool to free every human being from toil. It could also be the starting point of a large number of evil genie scenarios where we get what we asked for, in the form we least want it.
From a morality standpoint, we can't force AGIs to work for us. We also can't restrict their ability to self-evolve.
If we can resolve those conflicts in such ways we can coexist in peace with an intelligence that'll in all likelihood quickly surpass ours and partner with it, I'm in. If we build it and we can't resolve that, our opinion doesn't really matter.