Interestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost no general optimization task that doesn't (with a sufficiently long lookahead) involve taking over the world as an intermediate step.
(Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)
(Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)