Sort of a tangent, but if you remember AI Dungeon when it first came out, it was based on a fine-tuned GPT-2 checkpoint which of course was pre-RLHF. While its adherence to user prompts wasn’t fantastic, it got a lot of praise for its diversity at the time.
I almost wonder if you could do something like this: have a non-RLHF model that gets the user’s request and just spits out a cloud of possibilities. It might not be the most tightly tuned, but then you could have a RLHF model take that output and use it as a sort of quasi-seed to increase diversity, while still retaining the RLHF advantages of producing an ultimate response that aligns with the user’s initial request.
Sort of a tangent, but if you remember AI Dungeon when it first came out, it was based on a fine-tuned GPT-2 checkpoint which of course was pre-RLHF. While its adherence to user prompts wasn’t fantastic, it got a lot of praise for its diversity at the time.
I almost wonder if you could do something like this: have a non-RLHF model that gets the user’s request and just spits out a cloud of possibilities. It might not be the most tightly tuned, but then you could have a RLHF model take that output and use it as a sort of quasi-seed to increase diversity, while still retaining the RLHF advantages of producing an ultimate response that aligns with the user’s initial request.
https://en.wikipedia.org/wiki/AI_Dungeon