Comment by thomasahle

It would be the same if the model was "raw", trained only on text completion. But all models these days are RLHF'ed on (prompt, answer) pairs, so unfortunately they can get confused if the prompt already contains part of an answer.