Comment

Comment by thewopr

I'd argue, they aren't doing something future-proof right now because the fundamental architecture of the LLM makes it nearly impossible to guarantee the model will correctly respond event to special [system] tokens.

In your SQL example, the interpreter can deterministically distinguish between "instruct" and "data" (assuming proper escape obviously). In the LLM sense, you can only train the model to pick up on special characters. Even if [system] is a special token, the only reason the model cares about that special token is because it has been statistically trained to care, not designed to care.

You can't (??) make the LLM treat a token deterministically, at least not in my understanding of the current architectures. So there may always be an avenue for attack if you consume untrusted content into the LLM context. (At least without some aggressive model architecture changes).

parent thewopr 3 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton