Short version - I had omp talking to Qwen 3.8 27B on a pair of 3090s and every turn took way too long. Over a day, I (with Claude's help) logged the traffic and it was mostly the model thinking. Qwen 3.8 goes to xhigh reasoning if you don't tell it otherwise, and a few of omp's internal helpers don't tell it. A thinking budget and an effort level per role got turns significantly.
A couple of other things turned up along the way. vLLM was returning a truncated tool call as if it finished (half a file on disk, "wrote successfully"), and Bifrost was shuffling the tool definition fields between requests so the prefix cache missed. I have PRs (now merged) with Bifrost and mcp-go for the shuffling.
It was a very interesting learning experience about model serving and harnesses. This post really only covers the latter, but I'll do another on the vLLM side soon.
A couple of other things turned up along the way. vLLM was returning a truncated tool call as if it finished (half a file on disk, "wrote successfully"), and Bifrost was shuffling the tool definition fields between requests so the prefix cache missed. I have PRs (now merged) with Bifrost and mcp-go for the shuffling.
It was a very interesting learning experience about model serving and harnesses. This post really only covers the latter, but I'll do another on the vLLM side soon.