stratos123

Looking at the cited-by I found a paper that evaluated a bunch of such methods by LLM-judged novelty, https://arxiv.org/abs/2606.12790 . CD is weirdly weak according to these results, they get better changes with other methods - but all of the others are less technical and basically amount to asking the model to try harder at being original. I wonder whether CD would work for a task like programming...

The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6

I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of possible completions which are quite different and yet similarly probable, and if a base model is well-calibrated on that space, that lets you sample from that entire space and get many diverse websites. But an RL-trained model is miscalibrated and chooses a much smaller subspace of completions to assign unnaturally high logprobs to, so no matter how you sample, all the websites you get will look the same. It's not obvious to me why this happens; I wouldn't be surprised if, for example, a new optimization algorithm gets discovered that prevents RLHF from having such effects.

As for studies that measure this effect, I just found these: https://arxiv.org/abs/2505.00047 , https://arxiv.org/abs/2507.20956 , https://arxiv.org/abs/2604.16027