I believe the censorship is at the semantic level, not the token level. Same way RL allows training model responses independent of eventual input/output languages.
I’m sure the goal is to remove stuff in pre training, but it is sufficient to RL it away. Same way OpenAI models doubtlessly have training data relating to bio weapons or pedophilia, but it is pretty effectively suppressed via RL.
I’m sure the goal is to remove stuff in pre training, but it is sufficient to RL it away. Same way OpenAI models doubtlessly have training data relating to bio weapons or pedophilia, but it is pretty effectively suppressed via RL.