> mean, as pointed out by a sibling comment, the reason it's so hard to shut those things down is that they benefit a lot of people and there's huge organic demand. Even the morality is hotly debated, there's no absolute consensus on the badness of those things
And mine is that this can also be true of a misaligned AI.
It doesn't have to be like Terminator, it can be slowly doing something we like and where we overlook the downsides until it's too late.
Doesn't matter if that's "cure cancer" but the cure has a worse than cancer side effect that only manifests 10 years later, or if it's a mere design for a fusion reactor where we have to build it ourselves and that leads to weapons proliferation, or if it's A/B testing the design for a social media website to make it more engaging and it gets so engaging that people choose not to hook up IRL and start families.
> But, what's new? That's the history of humanity. The communists, the Jacobins, the Nazis, all thought they were building a better world and had to have their "off switch" thrown at great cost in lives.
Indeed.
I would agree that this is both more likely and less costly than "everyone dies".
But I'd still say it's really bad and we should try to figure out in advance how to minimise this outcome.
> except the misaligned intelligences are called academics
Well, that's novel; normally at this point I see people saying "corporations", and very rarely "governments".
Not seen academics get stick before, except in history books.
And mine is that this can also be true of a misaligned AI.
It doesn't have to be like Terminator, it can be slowly doing something we like and where we overlook the downsides until it's too late.
Doesn't matter if that's "cure cancer" but the cure has a worse than cancer side effect that only manifests 10 years later, or if it's a mere design for a fusion reactor where we have to build it ourselves and that leads to weapons proliferation, or if it's A/B testing the design for a social media website to make it more engaging and it gets so engaging that people choose not to hook up IRL and start families.
> But, what's new? That's the history of humanity. The communists, the Jacobins, the Nazis, all thought they were building a better world and had to have their "off switch" thrown at great cost in lives.
Indeed.
I would agree that this is both more likely and less costly than "everyone dies".
But I'd still say it's really bad and we should try to figure out in advance how to minimise this outcome.
> except the misaligned intelligences are called academics
Well, that's novel; normally at this point I see people saying "corporations", and very rarely "governments".
Not seen academics get stick before, except in history books.