These stories about AIs taking over the world in order to maximize a score often contain some computer that escapes the lab or such, in order to maximize some score. And indeed, it is true that taking over the world helps you to maximize whatever score you want to maximize.
But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain?
We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. And we don't realize it because economic growth seems normal.
These clippy stories speak of a not unfounded fear that development might not stop with us, that one day some life form spawns that is even smarter than us, and turns us, the predators, into irrelevant side pieces that need to get out of the way for fulfilling the objective of the more powerful life form, even if it was us who brought that life form into existence or gave it that objective.
The forests of today are not inhabited by the life form that first discovered photosynthesis, but instead by highly specialized structures that grow higher so that they can put their competition into the shade. Similarly, just because we are the first highly intelligent + cooperative life doesn't mean we will be the top of the world forever.
I'm not saying this with a light heart, I do feel sorry for the future humans, some of whom might even be alive today. But "how to contain reward optimizers" is a tough question, especially due to us running a giant project of reward optimization ourselves.
But mankind, as a whole, is a clippy optimizer already, albeit a manual one. Right now it is us that is destroying life around us, replacing rainforest with cattle farms and suburban developments. We are already working on farming robots to serve our goals. Does the lizard care if it is crushed by a farming robot's wheel or an AI tank's chain?
We are not the strongest, nor the biggest, nor the fastest, nor the best hearing, or the best seeing species. Still, we are the apex predator of this planet, thanks to our sole distinguishing feature, which is our intelligence and ability to cooperate. And we don't realize it because economic growth seems normal.
These clippy stories speak of a not unfounded fear that development might not stop with us, that one day some life form spawns that is even smarter than us, and turns us, the predators, into irrelevant side pieces that need to get out of the way for fulfilling the objective of the more powerful life form, even if it was us who brought that life form into existence or gave it that objective.
The forests of today are not inhabited by the life form that first discovered photosynthesis, but instead by highly specialized structures that grow higher so that they can put their competition into the shade. Similarly, just because we are the first highly intelligent + cooperative life doesn't mean we will be the top of the world forever.
I'm not saying this with a light heart, I do feel sorry for the future humans, some of whom might even be alive today. But "how to contain reward optimizers" is a tough question, especially due to us running a giant project of reward optimization ourselves.