Is the type returned guaranteed to be correct, or are you relying on non-hallucinations?
To be specific - to guarantee correctness you can reject logits for tokens that when used would produce an invalid JSON value returned from the LLM. For example if the string so far is
{ rating:
If the suggested token is something like "3
Then you can reject, because the next character needs to be 0-9, and re-sample from the distribution.
To be specific - to guarantee correctness you can reject logits for tokens that when used would produce an invalid JSON value returned from the LLM. For example if the string so far is
If the suggested token is something like "3Then you can reject, because the next character needs to be 0-9, and re-sample from the distribution.