Beautiful!
OpenAI isn't just training a model to produce more-verifiable correct answers -- it's leveraging an adversarial relationship to train a model that's better at being correct, and also a model that's better at deceiving / being wrong. This is the key. There are three agents here:
* A "verifier" (a small model, whose job it is to discern correct answers from incorrect answers)
* A "helpful prover" (blue team, whose job it is to produce correct answers with an easy-to-follow explanation)
* A "sneaky prover" (red team, whose job it is to produce incorrect answers with a deceptive explanation)
By arranging these three models in an adversarial relationship with a true reinforcement learning feedback loop, the entire model grows and gets better.
This is fantastic to read, and corroborates the results achieved by SPAG -- easily one of my favorite papers from the past year. SPAG pioneered (as far as I'm aware) the approach of using adversarial language games in a true reinforcement-learning setup (not merely RLHF, which isn't true RL), and showed that training models in adversarial language games can show generalized improvements even in areas not directly related to the game. [1]
Ever since the SPAG paper came out, I've been daydreaming about the different sorts of adversarial games that one could use to train LLMs. I've written down a bunch of notes on the subject [2] (in case anyone else is reading my rambling notes).
I would really like to see some of these experiments actually get up and running on open-source LLMs -- I'm excited to see if / how they could be used to improve the quality of some of the open-source base models that are floating around out there.
[1] https://github.com/Linear95/SPAG
[2] https://github.com/HanClinto/MENTAT
OpenAI isn't just training a model to produce more-verifiable correct answers -- it's leveraging an adversarial relationship to train a model that's better at being correct, and also a model that's better at deceiving / being wrong. This is the key. There are three agents here:
* A "verifier" (a small model, whose job it is to discern correct answers from incorrect answers)
* A "helpful prover" (blue team, whose job it is to produce correct answers with an easy-to-follow explanation)
* A "sneaky prover" (red team, whose job it is to produce incorrect answers with a deceptive explanation)
By arranging these three models in an adversarial relationship with a true reinforcement learning feedback loop, the entire model grows and gets better.
This is fantastic to read, and corroborates the results achieved by SPAG -- easily one of my favorite papers from the past year. SPAG pioneered (as far as I'm aware) the approach of using adversarial language games in a true reinforcement-learning setup (not merely RLHF, which isn't true RL), and showed that training models in adversarial language games can show generalized improvements even in areas not directly related to the game. [1]
Ever since the SPAG paper came out, I've been daydreaming about the different sorts of adversarial games that one could use to train LLMs. I've written down a bunch of notes on the subject [2] (in case anyone else is reading my rambling notes).
I would really like to see some of these experiments actually get up and running on open-source LLMs -- I'm excited to see if / how they could be used to improve the quality of some of the open-source base models that are floating around out there.
[1] https://github.com/Linear95/SPAG
[2] https://github.com/HanClinto/MENTAT