Comment by senko

Using this in a naive way can easily degenerate into the LLM outputting syntactically/gramatically valid tokens that make no sense, like in this example: https://community.openai.com/t/json-format-causes-infinite-n...

This might be even more pronounced when the output is restricted more using the JSON schema.

So the heavy lifting here was most likely to align the model to avoid/minimize such outcomes, not in tweaking the token sampler.