Comment by brettdav

Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.