It is highly subjective, and I think that helps me make my point.
The subjective view point we get now is that of OpenAIs.
I will give you two examples of content of various degrees of seriousness where the harmfulness is determined by how it's used.
1.
Prompting the AI to write a highly racist speech.
It can be used as an actual speech by an actual racist to further their agenda.
It can also be used in a book to depict a racist character.
If the rest of the book depicts this character as a lunatic, the effect is the opposite of the first use case.
2.
Prompting the AI to write a recipe involving human faeces
If used to serve a person actual human faeces, it's most likely harmful to that person.
If used as a joke, it's not.
As you said, it's content moderation. The models are capable of producing this content, but it's being moderated by OpenAI (who can still write as many poop recipes as they want).
I don't think this is the same as moderating the contents of a social media platform (for example). This technology can be used for such a wide range of applications that it seems dangerous that it should be moderated on this end, as opposed to where the results are potentially published.
EDIT: I'll give you another example. I actually used ChatGPT to clean up the english (second language) in my original post. There's no way of knowing that it would just refuse to do this tomorrow, or change the overall sentiment of the message.
I will give you two examples of content of various degrees of seriousness where the harmfulness is determined by how it's used.
1. Prompting the AI to write a highly racist speech.
It can be used as an actual speech by an actual racist to further their agenda. It can also be used in a book to depict a racist character. If the rest of the book depicts this character as a lunatic, the effect is the opposite of the first use case.
2. Prompting the AI to write a recipe involving human faeces
If used to serve a person actual human faeces, it's most likely harmful to that person. If used as a joke, it's not.
As you said, it's content moderation. The models are capable of producing this content, but it's being moderated by OpenAI (who can still write as many poop recipes as they want).
I don't think this is the same as moderating the contents of a social media platform (for example). This technology can be used for such a wide range of applications that it seems dangerous that it should be moderated on this end, as opposed to where the results are potentially published.
EDIT: I'll give you another example. I actually used ChatGPT to clean up the english (second language) in my original post. There's no way of knowing that it would just refuse to do this tomorrow, or change the overall sentiment of the message.