Yeah, no. LLMs do fantastic on problems they can copy out of their training set, but they fall apart on actually novel tasks. That's the entire reason we have to keep inventing new benchmarks, and why scores on new benchmarks are so low.
LLMs copy paste from their training set. That's plainly obvious when you try to do real work with them.
LLMs copy paste from their training set. That's plainly obvious when you try to do real work with them.