The relation to current RLVR methods I think is interesting, they do discuss it a bit but I would be curious to see more about this as well. Quote from the paper:
Exploration beyond Pretraining. The mode collapse XMs address during pretraining also often
shows up in post-training, where RL fine-tuning is known to sharpen models onto a narrow set of
behaviors [76]. Recent fixes such as pass@krewards [77] and best-of-N-aware fine-tuning [78] can
be seen through our lens as Forward XM, with a verifier standing in for ground truth data. These
fixes act only during post-training, though; pretraining with exploration may yield base models that
capture more modes in the first place, leaving RL more to select among.
Exploration beyond Pretraining. The mode collapse XMs address during pretraining also often shows up in post-training, where RL fine-tuning is known to sharpen models onto a narrow set of behaviors [76]. Recent fixes such as pass@krewards [77] and best-of-N-aware fine-tuning [78] can be seen through our lens as Forward XM, with a verifier standing in for ground truth data. These fixes act only during post-training, though; pretraining with exploration may yield base models that capture more modes in the first place, leaving RL more to select among.