Comment by jubilee33

But if the machines capability is beyond the ability of humans to evaluate, how would it make sense that humans are choosing the goals? The machine is more intelligent than you and thus logically able to solve problems and evaluate your needs better that you yourself are. The fact that this sounds incredibly scary should probably stop people from building said machine, but it won't. Some people, perhaps more than you would think, want control to pass out of humanity's hands, no matter the consequences. The Nash equilibrium of the situation kind of shows one outcome,that this will be built providing it's technically possible, we just need to kind of pray the outcome is utopia rather than oblivion. Not sure the odds honestly.