Comment by frogamel

I might be miunderstanding something here, but what complexity here is resolved by making this a framework? Isnt this just:

1. Train model like normal

2. Evaluate model using self

3. Use eval results for DPO finetune