Comment Comment by danielhanchen Oh sorry - did not expect to restate what you said whoops - all train of thought!You can see from https://huggingface.co/mobiuslabsgmbh/Llama-2-7b-chat-hf_1bi... that the model disk space is 3GB + 100MB of LoRA weights. I also uploaded a 4bit one to https://huggingface.co/unsloth/llama-2-7b-bnb-4bit/tree/main which uses 3.87GB.So because of the offloading trick, the GPU VRAM is less, but in actuality, still 3GB is needed.Unsure on latency sadly parent danielhanchen 2 years ago
You can see from https://huggingface.co/mobiuslabsgmbh/Llama-2-7b-chat-hf_1bi... that the model disk space is 3GB + 100MB of LoRA weights. I also uploaded a 4bit one to https://huggingface.co/unsloth/llama-2-7b-bnb-4bit/tree/main which uses 3.87GB.
So because of the offloading trick, the GPU VRAM is less, but in actuality, still 3GB is needed.
Unsure on latency sadly