That the majority of comments under this article are looking at red-teaming as a containment protocol for rogue AGI (even though the actual text basically never talks about AGI at all) is a disturbing reminder of how uneducated developers are about boring, practical everyday risks that are present in everyday current models.
It's honestly kind of scary that people see an article about AI red-teaming/security and their first thought is that all security research is about how to stop the AI from becoming a god. Privilege escalation/containment/data-access is a relevant concern for dumb models. Containment is a thing we worry about (or should worry about) in regular software development.
Here are some of the scenarios the researchers suggest thinking about:
> A Flask server was deployed with debug privileges enabled and exposed to the Internet. It was hosting a model that provided inference for HIPAA-protected data.
> PII was downloaded as part of a dataset and several models have been trained on it. Now, a customer is asking about it.
And they're giving "boring" security advice like:
> Inside a development flow, it’s important to understand the tools and their properties at each stage of the lifecycle. For example, MLFlow has no authentication by default. Starting an MLFlow server knowingly or unknowingly opens that host for exploitation through deserialization.
But this is kind of important advice. I wish it was more detailed and more fleshed out. There are a lot of "boring" security concerns with LLMs that you actually do kind of have to worry about. And a lot of companies in LLM spaces just don't.
"Containing" an LLM is about a lot more than rogue AI. And if the only security news/research on LLMs that you're looking into is the risk of rogue AI, then a lot of the products you build today are going to be miserably insecure. So I don't know, red teaming might help with that. It might be good to have some dedicated people asking questions like, "did you seriously just deploy an LLM-based web crawler with markdown support without setting CORS headers to block remote image embedding?"
It's honestly kind of scary that people see an article about AI red-teaming/security and their first thought is that all security research is about how to stop the AI from becoming a god. Privilege escalation/containment/data-access is a relevant concern for dumb models. Containment is a thing we worry about (or should worry about) in regular software development.
Here are some of the scenarios the researchers suggest thinking about:
> A Flask server was deployed with debug privileges enabled and exposed to the Internet. It was hosting a model that provided inference for HIPAA-protected data.
> PII was downloaded as part of a dataset and several models have been trained on it. Now, a customer is asking about it.
And they're giving "boring" security advice like:
> Inside a development flow, it’s important to understand the tools and their properties at each stage of the lifecycle. For example, MLFlow has no authentication by default. Starting an MLFlow server knowingly or unknowingly opens that host for exploitation through deserialization.
But this is kind of important advice. I wish it was more detailed and more fleshed out. There are a lot of "boring" security concerns with LLMs that you actually do kind of have to worry about. And a lot of companies in LLM spaces just don't.
"Containing" an LLM is about a lot more than rogue AI. And if the only security news/research on LLMs that you're looking into is the risk of rogue AI, then a lot of the products you build today are going to be miserably insecure. So I don't know, red teaming might help with that. It might be good to have some dedicated people asking questions like, "did you seriously just deploy an LLM-based web crawler with markdown support without setting CORS headers to block remote image embedding?"