Comment Comment by omneity 128GB but it's not using much.I'm running Q4 and it's taking 17.94 GB VRAM with 4k context window, 20GB with 32k tokens. parent omneity 1 year ago
I'm running Q4 and it's taking 17.94 GB VRAM with 4k context window, 20GB with 32k tokens.