The conclusion that numerate people are coming to regarding algorithms for court sentencing is quite different from what you describe.
The conclusion there is simply a theorem: it's mathematically impossible to be both well calibrated (a black and white person receiving the same risk score have the same probability of committing crime) and also racially balanced (similar levels of false positives), except in trivial and unrealistic cases.
But the thing is, this theorem applies to any decision process - human or algorithm. You can't escape mathematical impossibility results just by using humans (e.g. NP Complete problems are still hard even if done by humans).
Human processes just add additional bias, e.g. the editorial slant that Vox wants Facebook to add, or the (alleged) bias that Facebook's human editors added before Facebook went all algorithmic.
The conclusion there is simply a theorem: it's mathematically impossible to be both well calibrated (a black and white person receiving the same risk score have the same probability of committing crime) and also racially balanced (similar levels of false positives), except in trivial and unrealistic cases.
https://arxiv.org/pdf/1609.05807v1.pdf
See also this WaPo article explaining in simpler terms: https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1...
But the thing is, this theorem applies to any decision process - human or algorithm. You can't escape mathematical impossibility results just by using humans (e.g. NP Complete problems are still hard even if done by humans).
Human processes just add additional bias, e.g. the editorial slant that Vox wants Facebook to add, or the (alleged) bias that Facebook's human editors added before Facebook went all algorithmic.