This rhetoric (and it is rhetoric, in that it is designed to persuade) too often lacks nuance. Generating a 3 minute stand alone video is impressive, but it is a different task from generating code that must be consistent with an existing mountain of code. Generating code that implements a straightforward feature within an existing system is different to generating code that must make architectural decisions in novel domain or with a novel architecture. The Navier-Stokes result is impressive but 1) they spent a bajillion tokens / currency units to achieve it and 2) AFAIK all the recent mathematics results have been counterexamples, which are a different beast to proofs.
My own experience, pootling around with Claude like a peasant, is that it is simultaneously amazing at some tasks and an absolute moron at others. The more abstract the reasoning required, the worse it is.
My own experience, pootling around with Claude like a peasant, is that it is simultaneously amazing at some tasks and an absolute moron at others. The more abstract the reasoning required, the worse it is.