It's latent but not like impossible to comprehend latent. No?
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.