I think a better way to think about this might not be that this chatbot isn't dangerous, but the fact that this was developed under capitalism, an an organization that's ultimate goal is profitability, means that the incentives of the folks who built it (hella $) are baked into the underlying model, and there's a glut of evidence that profit-aligned entities (like businesses) are not necessarily (nor, I would argue, /can they be/) human-aligned.
This is the same as the facial-recognition models that mis-identify folks of color more frequently than white folks or the prediction model that recommended longer jail/prison sentences for black folks than for white folks who committed the same crime.
This is the same as the facial-recognition models that mis-identify folks of color more frequently than white folks or the prediction model that recommended longer jail/prison sentences for black folks than for white folks who committed the same crime.