I've had a very similar issue happening randomly. same pattern of the model outputting gibberish, seemingly trying to end its response and starting to panic when it couldn't, repeating a bunch of times.
turned out to be a bug in a patch, causing kv cache indices to be stored in too narrow variable types, making them wrap around.
fascinating, and slightly horrifying, that LLMs are able to notice that their output isn't what they meant to output. reminds me of the mirror test.
turned out to be a bug in a patch, causing kv cache indices to be stored in too narrow variable types, making them wrap around.
fascinating, and slightly horrifying, that LLMs are able to notice that their output isn't what they meant to output. reminds me of the mirror test.