Comment by jda5

I wonder if there is some bias creeping into the reseachers' methodology. Their paper replicates an experiment published in 2024, and depending on OpenAI's sampling, the original paper may have been part of GPT-5's training data. If so, then the LLM would have had exposure to both the questions and answers, biasing the model to choose the correct ones.