Comment by puzzlingcaptcha

Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.