I love your detailed explanation of things that I had little to no knowledge of, but I’m also stuck wondering if I’m using my setup correctly, as I have 16 GB of VRAM available but also 64 GB of regular RAM in my computer. Every time I use some setup or some configuration that overlaps or overflows between the two RAMs, my token per second drops significantly, me thinking that it is not set up correctly, so I’m stuck with 9 billion parameter models myself, even though I was under the impression with 64 GB of memory that I would be able to run something like a 30 billion model.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.