The problem with black boxes is that they beg parallel construction theories.
I.e. "We don't approve loans to {ethnic majority} on purpose. We only use our models, which only rely on historical data." -> No proof, no case, no problem
The point of any AI law should be to put the onus on the model executor to explain how their model doesn't violate existing law.
You can use any model you want, but you can't point to its unexplainability as a free pass for ignoring protected classes.
And if you can't explain its results? Well, then you're opening yourself up to lawsuits.
I.e. "We don't approve loans to {ethnic majority} on purpose. We only use our models, which only rely on historical data." -> No proof, no case, no problem
The point of any AI law should be to put the onus on the model executor to explain how their model doesn't violate existing law.
You can use any model you want, but you can't point to its unexplainability as a free pass for ignoring protected classes.
And if you can't explain its results? Well, then you're opening yourself up to lawsuits.