Why clearly? You mean the tensor pre seed with deterministic pseudo random numbers? I don't have much experience in LLM training, but I do remember from minimization algorithms (like gradient descent) that it's better to start with (pseudo) random starting values and minimize from there. And seeing how I would like my results repeatable, seeding a pseudo random sequence seems logical.
I don't understand why these tensors specifically weren't changed during minimization (training) though. Maybe they were not activated, or a w part of an image decoder or something?
I don't understand why these tensors specifically weren't changed during minimization (training) though. Maybe they were not activated, or a w part of an image decoder or something?