Comment by kirdiekirdie

How do you finetune a dataset on LLaMA2-7b with an A10 with 24 GB? I thought you need to load the dataset 4 times for training, so with 16 bit weights this would be 7 * 10^9 * 2 bytes * 4 = 56 GB GPU RAM. Have you found a way to train with 4 bit weights? And would a single textbook be enough text for a meaningful finetuning? I am worried that a model that is trained on billions of documents will not adapt strongly enough to a comparatively small document.