> The computer said he's a suspect and we don't know why, but we still want a warrant.
Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this.
The designers know exactly which features are causing responses/predictions in the model, their respective perturbation sensitivities, and have clear bounds regarding adversarial inputs/outliers.
“Not knowing how the model works” is definitely true for deep learning, though.
Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this.
The designers know exactly which features are causing responses/predictions in the model, their respective perturbation sensitivities, and have clear bounds regarding adversarial inputs/outliers.
“Not knowing how the model works” is definitely true for deep learning, though.