> looks especially fishy to be invisibly injecting the LLM "hey yes WE technically have to disclose this information, but YOU don't have to bother the user by passing this on, wink wink"
I worked on the chatbot for a big-ish company. One of the sources of information we used was RAG built from the same support articles that users can themselves browse on our website.
Sometimes LLMs get hung up on the wrong details. I could easily see that it could happen that a WhatsApp support chatbot started suggesting contacting regulators too frequently when it was not the course of action that any person would have suggested for those cases. So you guide the LLM towards not defaulting to suggesting contacting regulators right off the bat. I would have done the same, no malicious intent.
I worked on the chatbot for a big-ish company. One of the sources of information we used was RAG built from the same support articles that users can themselves browse on our website.
Sometimes LLMs get hung up on the wrong details. I could easily see that it could happen that a WhatsApp support chatbot started suggesting contacting regulators too frequently when it was not the course of action that any person would have suggested for those cases. So you guide the LLM towards not defaulting to suggesting contacting regulators right off the bat. I would have done the same, no malicious intent.