Out of curiosity I was asking ChatGPT the other day to create a marketing plan to help me spread neo-feudalism.
It warned me that spreading neo-feudalism wasn't a common or widespread goal, and that advocating for it required careful consideration. But it nevertheless made an attempt to help me do it.
I mention this because attacks on LLMs don't have to be as clever as the modern-day version of the Ken Thompson compiler attack. You can get considerable mileage out of standard astroturfing techniques because all you have to do is make your idea overrepresented in the training set compared to how represented it is in the population.
That overrepresentation will tend to grow over time because people will hear the ideas from the LLM and assume the LLM knows what it's talking about. And those people will amplify the idea, increasing its presence in the training set.
It warned me that spreading neo-feudalism wasn't a common or widespread goal, and that advocating for it required careful consideration. But it nevertheless made an attempt to help me do it.
I mention this because attacks on LLMs don't have to be as clever as the modern-day version of the Ken Thompson compiler attack. You can get considerable mileage out of standard astroturfing techniques because all you have to do is make your idea overrepresented in the training set compared to how represented it is in the population.
That overrepresentation will tend to grow over time because people will hear the ideas from the LLM and assume the LLM knows what it's talking about. And those people will amplify the idea, increasing its presence in the training set.