> Why shouldn't a generic prompt produce a mix of generic-but-notably-different outputs, as would be the case if you sampled humans?
Yeah, I hear what you’re saying. I guess it depends on how we define “notably different.” As I mentioned elsewhere, if you ask a random group of 12 people to draw a house on a piece of paper, you’ll end up with almost identical results. Sure, there will be some minor differences but for the most part, it’s going to be the same square with a triangular roof, two windows, one door, and maybe a chimney.
> AFAIK, the answer is that it's RLHF that causes models to collapse to a specific style, because base models tend to score much better on creativity and output diversity evaluations than post-trained ones.
Interesting. I'd love to see a paper on this. I wouldn't mind trying some back-of-the-envelope tests on a RHLF vs Base version of a locally hostable Qwen model too!
Yeah, I hear what you’re saying. I guess it depends on how we define “notably different.” As I mentioned elsewhere, if you ask a random group of 12 people to draw a house on a piece of paper, you’ll end up with almost identical results. Sure, there will be some minor differences but for the most part, it’s going to be the same square with a triangular roof, two windows, one door, and maybe a chimney.
> AFAIK, the answer is that it's RLHF that causes models to collapse to a specific style, because base models tend to score much better on creativity and output diversity evaluations than post-trained ones.
Interesting. I'd love to see a paper on this. I wouldn't mind trying some back-of-the-envelope tests on a RHLF vs Base version of a locally hostable Qwen model too!