The distinction here is between 'the /kind/ of data you're throwing at it' and 'data you have already thrown at it.'
The result shows that the LLM has a hard time generalizing to new (basic!) problems that were not in its training set; this suggests it is memorizing solutions rather than understanding the mechanisms... It's the undergrad who has crammed Leetcode solutions for two weeks before their interview, rather than a student with a deeper understanding who can successfully address new problems.
Having a friend who is objectively pretty dumb but has memorized the entire existing literature of your field of study is still pretty damned useful, however. You just need to understand what kinds of questions you can trust them with.
Along these lines, my own best use of LLMs is looking up jargon and old results. I see some complicated problem, propose a method for attacking it, and ask my friend who has read every stats paper ever digitized whether they have seen this kind of analysis before, what it's called, why it's a bad idea, and what people do instead. The answers point me to relevant literature by giving me the right jargon and method names to search for using Good-Old-Fashioned-Google.
The result shows that the LLM has a hard time generalizing to new (basic!) problems that were not in its training set; this suggests it is memorizing solutions rather than understanding the mechanisms... It's the undergrad who has crammed Leetcode solutions for two weeks before their interview, rather than a student with a deeper understanding who can successfully address new problems.
Having a friend who is objectively pretty dumb but has memorized the entire existing literature of your field of study is still pretty damned useful, however. You just need to understand what kinds of questions you can trust them with.
Along these lines, my own best use of LLMs is looking up jargon and old results. I see some complicated problem, propose a method for attacking it, and ask my friend who has read every stats paper ever digitized whether they have seen this kind of analysis before, what it's called, why it's a bad idea, and what people do instead. The answers point me to relevant literature by giving me the right jargon and method names to search for using Good-Old-Fashioned-Google.