jesdo

We currently focus on serving DeepSeek models on dedicated clusters for large customers. We do not offer this as a publicly available service at the moment, but we may consider doing so in the future.

I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.

Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.

Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.

Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.

To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.

I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.

I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)

Great initiative, and good to see that it has an effect. I'm a bit sceptical about the available funds. 1 million over 5 years is a nice starting package (4+ PhD students), but the availability of overall research grant money in the Netherlands has been under pressure for years and is difficult to acquire. Researchers moving here may find it difficult to acquire further grant money compared to US, at least in CS.