> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal.
This is incidental to the main disagreement, but btw I also doubt this.
Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of full-time work (without AI assistance obviously)? I'd bet against it!
This is incidental to the main disagreement, but btw I also doubt this.
Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of full-time work (without AI assistance obviously)? I'd bet against it!
Here is the manual for the cal that we use: https://media.githubusercontent.com/media/epoch-research/Mir...
You can also look at a full transcript of an LLM solving the task: https://epochai-public-eval-logs-manual.s3.amazonaws.com/eva...
The data is here: https://github.com/epoch-research/MirrorCode-data/