Comment by roger_

I guess “A” by itself would be a seperate token but my point is the model might be trying to say something that begins with that letter rather than actually answering.

You’d need to use an approach that links the output to a closed set.