Arguably this is already happening. Governments and transnational corporations are "AIs" in the sense that their macro-level behavior is largely determined by systematic structure and incentives, not individual decision-making. No individual human has influence over the whole, and those at individual points in the system are incentivized only to fulfill their particular role instead of look out for the system in general.
Capitalism itself, can be viewed as just a giant paperclip maximizer.
Capitalism itself, can be viewed as just a giant paperclip maximizer.