I'm confused - it seems to be saying that, instead of thinking about neurons as packing together a lot of "concepts" together into one, the issue is that your view of what a "concept" is, was already prejudiced, and the neuron itself had a small clean role? Like, you can either think about this as the "Cat & Car" neuron or you can think about it as the "V shaped neuron" - one seems simpler.
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.