Comment by sersi

I mean it is a bug, if they had been smart about it, they'd just have only added the instructions to the prompt or have their chatbot get a slightly different help page with the instructions embedded in there and not made it public.

Implementing it this way for their own chatbot is stupid. Now as to whether it's ethical or not that's something else.