> which is equivalent to just finding the most similar key and then returning the corresponding value. Why would you take a combination of all powers, when you know that higher powers work better?
The article has the answer. The problem with that is that it is no longer differentiable.
The limit to infinite powers is to use the exponential function, which has weighted powers up to infinity. But you still need a kind of smoothing factor so that you don't end up just picking the closest one and losing differentiability, which they call beta and is picked based on how many memories you've stored.
The article has the answer. The problem with that is that it is no longer differentiable.
The limit to infinite powers is to use the exponential function, which has weighted powers up to infinity. But you still need a kind of smoothing factor so that you don't end up just picking the closest one and losing differentiability, which they call beta and is picked based on how many memories you've stored.