Comment by HarHarVeryFunny

Yes, although the sparsity doesn't need to be inherent to the model - another approach is to try to decode the learned weights using approaches like sparse auto-encoders or transcoders.

https://transformer-circuits.pub/2025/attribution-graphs/met...