That is not entirely true. At least some of these tests (like HLE and ARC) take steps to keep the evaluation set private so that LLMs can’t just memorize the answers.
You could question how well this works, but it’s not like the answers are just hanging out on the public internet.
You could question how well this works, but it’s not like the answers are just hanging out on the public internet.