Comment by lawlessone

From the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance.

This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power.

Is this only for inference though? could it boost training?