> Interesting. I'd love to see a paper on this.
The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6
I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of possible completions which are quite different and yet similarly probable, and if a base model is well-calibrated on that space, that lets you sample from that entire space and get many diverse websites. But an RL-trained model is miscalibrated and chooses a much smaller subspace of completions to assign unnaturally high logprobs to, so no matter how you sample, all the websites you get will look the same. It's not obvious to me why this happens; I wouldn't be surprised if, for example, a new optimization algorithm gets discovered that prevents RLHF from having such effects.
As for studies that measure this effect, I just found these: https://arxiv.org/abs/2505.00047 , https://arxiv.org/abs/2507.20956 , https://arxiv.org/abs/2604.16027
The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6
I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of possible completions which are quite different and yet similarly probable, and if a base model is well-calibrated on that space, that lets you sample from that entire space and get many diverse websites. But an RL-trained model is miscalibrated and chooses a much smaller subspace of completions to assign unnaturally high logprobs to, so no matter how you sample, all the websites you get will look the same. It's not obvious to me why this happens; I wouldn't be surprised if, for example, a new optimization algorithm gets discovered that prevents RLHF from having such effects.
As for studies that measure this effect, I just found these: https://arxiv.org/abs/2505.00047 , https://arxiv.org/abs/2507.20956 , https://arxiv.org/abs/2604.16027