Comment Comment by cjbprime Hm, I have a 4090 as well, and:$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memoryUpdate: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail? parent cjbprime 2 months ago
$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory
Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?