You are both talking about this claim?
LLMs do fantastic on problems they can copy out of their training set, but they fall apart on actually novel tasks.
I mean, I give Claude what I consider novel problem solving work all the time.
You can make the statement true if you raise the bar for "novel task" to a point where most humans also can't do them. In these threads the goal posts for words like "understanding", "creative", "novel" etc gets moved for LLMs to way past their previous definitions, in fact way past a large % of the worlds human software developers.
To just take one example: Ask for a security review and you get back a list of all the things you would have found yourself after some days of pouring over the code (if you are senior developer well versed in security issues), plus the ones you would have overlooked, but in 15 minutes instead of in several days.
Before LLMs came along this would obviously have been "novel work". A shining example of a human using their intellect! But now that LLMs can do it it is just "copy and paste work"...
You can make the statement true if you raise the bar for "novel task" to a point where most humans also can't do them. In these threads the goal posts for words like "understanding", "creative", "novel" etc gets moved for LLMs to way past their previous definitions, in fact way past a large % of the worlds human software developers.
To just take one example: Ask for a security review and you get back a list of all the things you would have found yourself after some days of pouring over the code (if you are senior developer well versed in security issues), plus the ones you would have overlooked, but in 15 minutes instead of in several days.
Before LLMs came along this would obviously have been "novel work". A shining example of a human using their intellect! But now that LLMs can do it it is just "copy and paste work"...