Comment by mahoho

I think the machine code–level understanding is what's important. We can, in theory, put a person in a Chinese Room–style scenario and have them manually perform the code, and it will generate the same outputs (It would probably take millions or billions of years, but it is true in principle). A major difference is that we created the machine and the code and, at least as low as the level of digital logic design, we understand and control its behavior. The person in the room has a human mind with thoughts and behaviors completely out of the program designers' control and unrelated to the program; if they want to, they can break out of the room and punch the operator. The "unpredictability" of the machine is still constrained by the fundamental capabilities we give to it, so it might generate surprising outputs but it can't do things like punch people or launch nukes unless we connect it to other systems that have those capabilities.