May be worth pointing out, that this is not the first residual connection innovation to be in production.
Gemma 3n is also using a low-rank projection of the residual stream called LAuReL. Google did not publicize this too much, I noted it when poking around in the model file.
Gemma 3n is also using a low-rank projection of the residual stream called LAuReL. Google did not publicize this too much, I noted it when poking around in the model file.
https://arxiv.org/pdf/2411.07501v3
https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3...
Seems to be what they call LAuReL-LR in the paper, with D=2048 and R=64