Comment by astrange

An issue with using the same benchmark for everything is, you eventually start training on the test set by natural selection via people throwing out all their attempts that don't work.

I'm not sure how to prevent this, maybe you can preregister your programmers?