Comment by loandbehold

This is pretty much what underlies AI doomer argument of people like EY. Humans will gradually hand over civilization to black box AI they can't understand. As AI becomes more complex and powerful it will be harder and harder to control.