The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.
Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.
I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.
Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.
I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.