In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to influence eval/training methods).
Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.
Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.