requires all algorithms to "explain" their reasoning
It's about the application. If you are detecting cat images on a social network, then a few false positives or negatives are totally no big deal. If your algo is making investment decisions or is being used as evidence in court then it is entirely reasonable to expect it to have a rock solid causal chain easily comprehensible by a layperson. The example in the article is noise reduction and the method was "we tweaked it until we got the results we wanted". In many fields that's called "overfitting" (or "fitting up") and is a big red flag.
It's about the application. If you are detecting cat images on a social network, then a few false positives or negatives are totally no big deal. If your algo is making investment decisions or is being used as evidence in court then it is entirely reasonable to expect it to have a rock solid causal chain easily comprehensible by a layperson. The example in the article is noise reduction and the method was "we tweaked it until we got the results we wanted". In many fields that's called "overfitting" (or "fitting up") and is a big red flag.