contemporary343

Yep...

Hahahahaha

Have you ever… been in an actual lab..? There are more disciplines in the world than computer science, you know

Echoing what the other reply to your question said, I don't disagree that much of this post-hoc, but I think the thinking has evolved such that this post-hoc reasoning is now truly believed. I'm primarily speaking of it as an observer of how and where funds are being directed and to what ends: often big, non-university-centered programs to build labs or generate datasets for AI discovery. While some of the ideas may have merit, there is no free lunch unfortunately, as that was money that would have supported individual PIs and their trainees.

Finding some new combination or iteration in the literature and running DFT is the kind of thing a senior undergraduate or first year grad student typically does (and typically with Claude anyway these days). (And yes, they'd probably use Quantum Espresso to start, like this writeup and its agent does). They'd probably show it at a weekly lab meeting where it would get ripped apart. And they would not be blasting a preliminary calculation around the world as if they'd made a new discovery.. but hey, we're in a brave new world; maybe they should!

It's an evals platform. The problem is to promote evals in scientific domains you need to actually know something about them. Otherwise you end up with slop like this.

This is what happens when Claude writes it for you (and you don't review it)

They ran Quantum Espresso which is ok, but by no means the 'state of the art' for DFT. And in case, any DFT computation has to be taken with a few pounds of grains of salt before getting too excited about it.

No offense to the person writing this (assuming they did at all), but I'm not sure they really understand what they're doing..

I'm sorry but this would be a mediocre paper at best. And if this were a student presenting this for a qualifying exam, you can bet a committee would be ripping them a new one for presenting this with no understanding of what it does.

"I was shown a prompt and told the internal research model had simply been given the problem statement. Levent had been told by Sebastien “very little human input” had been used. This turned out not to be true. Over the course of the call, as members of their team sent Sebastien corrections and details over their internal chat, it emerged that an entire team had been working on the problem, that this was one of a number of things that was tried, that work had started on the unforced problem, that the team first set the model on easier problems, including Euler, that even the prompt that had been shown to me had been written by prompting Codex, and that an insane amount of compute had been used."

- This, from Tristan Buckmaster's writeup yesterday, indicates to me that there was more than incidental inspiration from Alpoge and Buckmaster.

"I was shown a prompt and told the internal research model had simply been given the problem statement. Levent had been told by Sebastien “very little human input” had been used. This turned out not to be true. Over the course of the call, as members of their team sent Sebastien corrections and details over their internal chat, it emerged that an entire team had been working on the problem, that this was one of a number of things that was tried, that work had started on the unforced problem, that the team first set the model on easier problems, including Euler, that even the prompt that had been shown to me had been written by prompting Codex, and that an insane amount of compute had been used."

One of the interesting threads here that is certainly relevant to the OpenAI writeup is the human role in the process. Buckmaster clearly points out that (exceptional!) mathematicians at OpenAI were certainly involved in correcting and guiding the process - and that their path/strategy was no doubt influenced by Alpoge & Buckmaster's work. It is always in OpenAI's interest to de-emphasize the role of people in the process, as is clearly the case here. Indeed, given sufficient compute and resources, I suspect Buckmaster could have also extended their approach to N-S.