> I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)
Is there any other mechanism replacing it? As far as I understood, while following the LLMs from Scratch book, the dropout is a mechanism to prevent overfitting, and it makes a lot of sense (basically cripple some neuron connections randomly during training such that their weights aren't updated).
Edit: well, I could have searched on the internet, but other people may stumble on your post and you seem quite knowledgeable of the inner workings of these things. :)