We've already built things in computing that we don't easily understand, even outside of AI, like large distributed systems and all sorts of balls of mud.
Within the sphere of AI, we have built machines which can play strategy games like chess, and surprise us with an unforseen defeat. It's not necessarily easy to see how that emerged from the individual rules.
Even a compiler can surprise you. You code up some optimizations, which are logically separate, but then a combination of them does something startling.
Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space.
Within the sphere of AI, we have built machines which can play strategy games like chess, and surprise us with an unforseen defeat. It's not necessarily easy to see how that emerged from the individual rules.
Even a compiler can surprise you. You code up some optimizations, which are logically separate, but then a combination of them does something startling.
Basically, in mathematics, you cannot grasp all the details of a vast space just from knowing the axioms which generate it and a few things which follow from them. Elementary school children know what is a prime number, yet those things occupy mathematicians who find new surprises in that space.