In neuroscience, predictive coding [1] is a theory that proposes the brain makes predictions about incoming sensory information and adjusts them based on any discrepancies between the predicted and actual sensory input. It involves simultaneous learning and inference, and there is some research [2] that suggests it is related to back-propagation.
Given that large language models perform some kind of implicit gradient descent during in-context learning, it raises the question of whether they are also doing some form of predictive coding. If so, could this provide insights on how to better leverage stochasticity in language models?
I'm not particularly knowledgeable in the area of probabilistic (variational) inference, I realize that attempting to draw connections to this topic might be a bit of a stretch.
Given that large language models perform some kind of implicit gradient descent during in-context learning, it raises the question of whether they are also doing some form of predictive coding. If so, could this provide insights on how to better leverage stochasticity in language models?
I'm not particularly knowledgeable in the area of probabilistic (variational) inference, I realize that attempting to draw connections to this topic might be a bit of a stretch.
[1] The free-energy principle: a unified brain theory: <https://www.fil.ion.ucl.ac.uk/~karl/The%20free-energy%20prin...>
[2] Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?: https://arxiv.org/abs/2202.09467