Comment Comment by kelseyfrog > When you finetune with lora, you're updating maybe 5% of the parametersI'm not sure I understand this comment. The LoRA paper[1] specifically says that all of the pretrained weights remain frozen.> keeping the pre-trained weights frozenSpecifically, the LoRA paper differentiates itself from updating some parameters by stating> Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks.1. https://arxiv.org/pdf/2106.09685 parent kelseyfrog 2 years ago
I'm not sure I understand this comment. The LoRA paper[1] specifically says that all of the pretrained weights remain frozen.
> keeping the pre-trained weights frozen
Specifically, the LoRA paper differentiates itself from updating some parameters by stating
> Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks.
1. https://arxiv.org/pdf/2106.09685