That's the whole problem: systems aren't deliberately designed this way, but LLMs are incapable of reliably distinguishing the difference between instructions from their users and instructions that might have snuck their way in through other text the LLM is exposed to.
My original name for this problem was "prompt injection" because it's like SQL injection - it's a problem that occurs when you concatenate together trusted and untrusted strings.
Unfortunately, SQL injection has known fixes - correctly escaping and/or parameterizing queries.
My original name for this problem was "prompt injection" because it's like SQL injection - it's a problem that occurs when you concatenate together trusted and untrusted strings.
Unfortunately, SQL injection has known fixes - correctly escaping and/or parameterizing queries.
There is no equivalent mechanism for LLM prompts.