I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?