I’ve heard many people say that the unexplainable nature of AI/ML models makes them untrustworthy, but do developers really care in practice? From my experience it seems like all that matters is performance considerations (eg accuracy, data efficiency, model size, etc). And the field seems to have been progressing fine despite the “black box”.
What would ideal explainability even look like? For example, in deep learning if we knew what individual neurons were doing, would this solve any problems? Or even if we could specifically know why the model makes a particular error, the solution is still just to throw more training data at the model. Maybe increased explainability could help with more rational design of model architectures and training methods, but that has never been such a huge bottleneck in my experience.
SHAP, LIME, saliency maps, and the like I have always felt to be pretty hand wavy, inaccurate, and they don’t provide much insight for what developers need. I feel like they only exist to placate AI/ML skeptics so we can claim we sort of know what is going on with our models only when someone cares.
Am I missing something here? What specific efficiencies or insights would a developer benefit from if we had more explainable AI/ML models?
What would ideal explainability even look like? For example, in deep learning if we knew what individual neurons were doing, would this solve any problems? Or even if we could specifically know why the model makes a particular error, the solution is still just to throw more training data at the model. Maybe increased explainability could help with more rational design of model architectures and training methods, but that has never been such a huge bottleneck in my experience.
SHAP, LIME, saliency maps, and the like I have always felt to be pretty hand wavy, inaccurate, and they don’t provide much insight for what developers need. I feel like they only exist to placate AI/ML skeptics so we can claim we sort of know what is going on with our models only when someone cares.
Am I missing something here? What specific efficiencies or insights would a developer benefit from if we had more explainable AI/ML models?