Comment Comment by fluffyspork According to Gemini to run full 256k context window with unified RAM4-bit Quantized (Q4_K_M GGUF): You need at least 25 GB of total RAM. This is the most practical configuration for consumer hardware.8-bit Quantized (Q8_0 / SFP8): You need at least 32 GB to 36 GB of RAMUncompressed 16-bit (BF16): You will need upwards of 45 GB to 50 GB of RAM to account for both the 26.7 GB base model and the massive KV cache. parent fluffyspork 5 months ago
4-bit Quantized (Q4_K_M GGUF): You need at least 25 GB of total RAM. This is the most practical configuration for consumer hardware.
8-bit Quantized (Q8_0 / SFP8): You need at least 32 GB to 36 GB of RAM
Uncompressed 16-bit (BF16): You will need upwards of 45 GB to 50 GB of RAM to account for both the 26.7 GB base model and the massive KV cache.