There is a bug : While in SwiGLU beta is a learnable parameter, in the reference paper the feed forward network set beta as a constant FFnSwiGLU = Swish1...
https://arxiv.org/pdf/2002.05202.pdf (Eq 6.)
In the blog log we observe various lines " feedforward.1.beta', 0.0 " which mean that during the training the beta has degenerated into 0 whereas it should be constant 1.
In official llama implementation : the constant beta has been removed https://github.com/facebookresearch/llama/blob/main/llama/mo...
In the blog log we observe various lines " feedforward.1.beta', 0.0 " which mean that during the training the beta has degenerated into 0 whereas it should be constant 1.