Comment by mansoor_

Not really. For a trivial function fitting problem, a KAN will allow you to visualise the contribution of each base function into the next layer of your network. Still, these trivial shallow networks are the ones nobody needs to introspect. A deep NN will not be explainable using this approach.