The spread isn't as large as I thought from the headline. It's closer to a toss-up than I would have expected.
The length of the response is a huge factor:
> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.
So the score is partially a proxy for longest responses.
Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?
The length of the response is a huge factor:
> Students tended to prefer longer responses. The selected answer was 37% longer on average than the alternatives. The longest response won 47.7% of decisive writing comparisons. The shortest still won 25.0%.
So the score is partially a proxy for longest responses.
Makes me wonder how much the reviewers actually read the text. Were lazy evaluators picking the text that looked the longest or most structured without reading it all?