Comment by huyegn

That's an interesting idea and I get the motivation. I often wonder if we're already able explain predictions. There are things like gradcam to determine which inputs caused which activations. It's just that the explanations aren't very understandable or satisfying from a human perspective.