Comment by macwhisperer

pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...

I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak