A bit out of topic, but has there been any research done on learn-time topology optimisations? E.g. Start with 4-4-4-4 network, remove some little contributing nodes/connections during training, resulting in 4-3-2-4 network with similar accuracy as 4x4, but higher performance?
Intuitively I think that there should be connections activated for less than x% of inputs that could be removed entirely. In some cases this would mean removal of whole node. Would interesting to read something about such an approach.
Intuitively I think that there should be connections activated for less than x% of inputs that could be removed entirely. In some cases this would mean removal of whole node. Would interesting to read something about such an approach.