I think what he describes here is a fundamental flaw of human nature. We now have overwhelming evidence that humans are really good at taking incomplete, disparate pieces of information and coalescing them into a whole to reach a conclusion. We generally infer very well. The problem is that the conclusions we reach are also generally wrong. Given a very specific type of domain with deterministic answers (e.g. most engineering fields) we are able to reach more often than not correct answers. And the rate at which we are correct has only increased (nearing 100%). However, when it comes to the messy domains in which we live we are still pretty poor. Our instincts are generally good enough to keep us alive (don't walk down this dark alley, that guy looks shady, etc) but much more than that and we suck.
I for one am deeply uncomfortable with people training AI systems to form the kind of decisions we as humans do. These systems may have more information to go on but if it's using the same methods that we do, how accurate can it be? Thanks to the rapid advancement of the last 150 years we have quickly reached a place where society's technological knowledge and ability far exceeds our moral ability to reason about it, and it's implications. We are like an untrained child with a gun. Yeah we may understand the mechanics behind pulling the trigger and how it works but we don't yet have the maturity to understand when we should, and what truly happens when we do.
I for one am deeply uncomfortable with people training AI systems to form the kind of decisions we as humans do. These systems may have more information to go on but if it's using the same methods that we do, how accurate can it be? Thanks to the rapid advancement of the last 150 years we have quickly reached a place where society's technological knowledge and ability far exceeds our moral ability to reason about it, and it's implications. We are like an untrained child with a gun. Yeah we may understand the mechanics behind pulling the trigger and how it works but we don't yet have the maturity to understand when we should, and what truly happens when we do.