dougcalobrisi

Short version - I had omp talking to Qwen 3.8 27B on a pair of 3090s and every turn took way too long. Over a day, I (with Claude's help) logged the traffic and it was mostly the model thinking. Qwen 3.8 goes to xhigh reasoning if you don't tell it otherwise, and a few of omp's internal helpers don't tell it. A thinking budget and an effort level per role got turns significantly.

A couple of other things turned up along the way. vLLM was returning a truncated tool call as if it finished (half a file on disk, "wrote successfully"), and Bifrost was shuffling the tool definition fields between requests so the prefix cache missed. I have PRs (now merged) with Bifrost and mcp-go for the shuffling.

It was a very interesting learning experience about model serving and harnesses. This post really only covers the latter, but I'll do another on the vLLM side soon.

You’re right. I may borrow that if I do a follow up at some point :-)

It does take videos (like mp4) as input but will only output the stripped audio track.

I might add the custom filler word functionality and/or perhaps just make the filler word list configurable.

I'm a big fan of this pattern. It's helped me a lot recently to document a system via Claude poking at it and exploring, journaling as it went.