Comment by pcstl

I don't think you can say an AI trained using RLHF - such as ChatGPT is - is really "simply reflecting the data it was trained on". ChatGPT was first trained on a load of data, then it was updated to act in specific ways based on feedback from humans who "nudged" it the way they wanted it to go.