The A3B part of that model means that each "expert" in the mixture of experts (MOE) model use 3B parameters. At 4-bit quant, you can fit 2 parameters per byte. That gives you 1.5GB per expert (plus a bit for overhead), which makes it usable on an 8GB card.
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.