mayukhdeb

The explanation for the original title is this plot from our publication in ICLR 2025: https://toponets.github.io/webpage_assets/FigureEfficiencyNa...

You can find more details on the website: https://toponets.github.io (see section: "Toponets deliver sparse, parameter-efficient language models")

We find out that inducing topographic structure in the weights of GPTs made them compressible (during inference) without losing out on performance.

I encourage you to revert the name if you find it justified after looking into the evidence I've shown here. Thanks.

This is true. The features closer together now have much stronger semantic overlap. You can watch how the weights self-organize in a GPT here: https://toponets.github.io/webpage_assets/banner_video.mp4

We're already studying the effects of topographic structure on polysemanticity.

More work is being done on this as we speak.

1. Significantly lower dimensionality of internal representations 2. More interpretable (see: https://toponets.github.io)

> 7B model down to 6B

We remove ~80% of the parameters in topographic layers and retain the same performance in the model. The drop in parameter count is not significant because we did not experiment with applying TopoLoss in all of the layers of the model (did not align with the goal of the paper)

We are currently performing those strong sparsity experiments internally, and the results look very promising!