This study is informative and most likely directionally right: prompt injection remains an unsolved problem, and adversarial input can fool modern ML models.
But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?
But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?
I’m sure I’m missing something!