I agree with Yann here ... I think the demand for explainability is like a person asking for a "faster horse" when what they really need is a car.
When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates .
Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it.
For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know!
I've been doing a review of techniques that are becoming more popular in this area:
When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates .
Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it.
For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know!
I've been doing a review of techniques that are becoming more popular in this area:
https://blog.everyhue.me/posts/why-uncertainty-matters/