This argument really doesn't hold water to me. It seems dramatically
more likely to me that the human agent overriding the model is going to be prejudiced than the mathematical model itself.
Of course it's possible (in fact, almost certain) that a math model trained on a large set of data is going to pick up on some problematic features. However, is it really more likely that these statistical inferences are more biased than a human being?
I'm sorry, but in my experience the number of racist human beings outweighs the number of racist computers.
Your examples seem so fraught. The Johnsons are unreliable, from a human, seems as likely to mean that John Johnson and Mr. Overriding Agent's sister had a nasty breakup as it does to mean they're likely to bounce checks. The Kennedys are good for it just sounds like code for, The Kennedys are of the racial group Agent prefers.
I agree with you that we can't blindly follow computer models, but I don't think I follow you to your conclusion that the loan officer was a valuable safety net.
Of course it's possible (in fact, almost certain) that a math model trained on a large set of data is going to pick up on some problematic features. However, is it really more likely that these statistical inferences are more biased than a human being?
I'm sorry, but in my experience the number of racist human beings outweighs the number of racist computers.
Your examples seem so fraught. The Johnsons are unreliable, from a human, seems as likely to mean that John Johnson and Mr. Overriding Agent's sister had a nasty breakup as it does to mean they're likely to bounce checks. The Kennedys are good for it just sounds like code for, The Kennedys are of the racial group Agent prefers.
I agree with you that we can't blindly follow computer models, but I don't think I follow you to your conclusion that the loan officer was a valuable safety net.