We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf.
If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous.
An attacker only needs to get lucky with their prompt attacks once.
We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf.
If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous.
An attacker only needs to get lucky with their prompt attacks once.
I wrote about the larger threat introduced by these new applications here: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/