Just a note that you have to have at least 12GB VRAM for it to be worth even trying to use your GPU for LLaMA 2.
The 7B model quantized to 4 bits can fit in 8GB VRAM with room for the context, but is pretty useless for getting good results in my experience. 13B is better but still not anything near as good as the 70B, which would require >35GB VRAM to use at 4 bit quantization.
My solution for playing with this was just to upgrade my PC's RAM to 64GB. It's slower than the GPU, but it was way cheaper and I can run the 70B model easily.
The 7B model quantized to 4 bits can fit in 8GB VRAM with room for the context, but is pretty useless for getting good results in my experience. 13B is better but still not anything near as good as the 70B, which would require >35GB VRAM to use at 4 bit quantization.
My solution for playing with this was just to upgrade my PC's RAM to 64GB. It's slower than the GPU, but it was way cheaper and I can run the 70B model easily.