Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run.
In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.