I think more checks (both stronger user-intent checks and adversarial tests for spec compliance) are needed to ensure data-quality when constructing training environments, but not necessarily always good ideas as explicit reward signals (which then can be gamed in different ways).
In a well-constructed training environment, models would not be thinking about the concept of a grader nor recognize that this is an eval rather than production usage. I think about these more as desiderata for the environment itself, rather than as explicit reward-signals to shape the model.
But of course researchers should also simultaneously cut/patch shortcuts/exploits that we find in reward-signals (models should never be able to get full reward with an unsatisfactory solution).
In a well-constructed training environment, models would not be thinking about the concept of a grader nor recognize that this is an eval rather than production usage. I think about these more as desiderata for the environment itself, rather than as explicit reward-signals to shape the model.
But of course researchers should also simultaneously cut/patch shortcuts/exploits that we find in reward-signals (models should never be able to get full reward with an unsatisfactory solution).