Comment by yurimo

Scale is an interesting point here. Large overparametrized networks don't behave in the same way as small toy models do; this is one of the big caveats of many interp works. I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it. You could theoretically improve a model on some form of weight recognition; maybe it will work, maybe it won't. But again, we come up against a cost/benefit dilemma; it sounds exotic and sci-fi but likely impractical. Parameters of the model are connected and relate to each other. Instead, current SOTA auto-interp argues that what we need are better tools for models to know themselves, so rather than direct edit, we need to give models better understanding of large parameter geometries, then they apply edits with non-destructive approaches at scale. This is much harder to do than applying a steering vector.