Comment by shakewingo

I bet it's just one of the prompt injections that lots of AI models exists nowadays. There are some of them that we don't realize yet. The safety and alignment problem is getting more and more important as model capability increases.