You're speaking from the perspective of someone who has to be responsible for something without understanding it, which is another pathology endemic to that system. By only using black box reasoning, of course you come to the conclusion that you need many such boxen in case one malfunctions in an unforeseeable way.
The thing is, the level of intelligence to maintain the software has to exist somewhere. If it's not vested in a small number of people able to actually understand the system, then the intelligence ends up being an emergent property of the human automatons, Chinese-room style, with the now-important "manager" pretending to control it when in fact nobody does.
The thing is, the level of intelligence to maintain the software has to exist somewhere. If it's not vested in a small number of people able to actually understand the system, then the intelligence ends up being an emergent property of the human automatons, Chinese-room style, with the now-important "manager" pretending to control it when in fact nobody does.