What if I tell the model to go commit fraud or crimes and it complies? What if users are having psychotic episodes driven by their interactions with the model?
Just because safety is a hard and messy problem doesn't mean we should just wash our hands of it.
Just because safety is a hard and messy problem doesn't mean we should just wash our hands of it.