Whenever we're testing LLM's against people we need to ask "which people?" Testing a chess bot against random undergrads versus chess grandmasters tells us different things.
From an economics perspective, maybe a relevant comparison is to people who do that task professionally.
From an economics perspective, maybe a relevant comparison is to people who do that task professionally.