qbit42

make no mistakes

mostly jk, although agents themselves are of course getting better at detecting whether a formal spec is faithful to a natural language spec

For what it's worth, most mathematicians are paid to teach, and the amount of research funding going into mathematics is relatively small. Plus, there are rare but important broader advancements due to pure and applied math that are hard to predict in advance (e.g., cryptography, compressed sensing).

Mathematicians also decide what to prove in their research - taste is not a skill we have RLVR'd away (yet, not claiming this is fundamental).

Of course, the field is being fundamentally transformed, it will probably end up looking more like experimental science in the future.

In his recent interviews, he says he has very little time to do math these days outside of research meetings with his PhD students and postdocs.

Is the use of "generalization" like this standard in mech interpretability? Seems undesirable given the typical definition in ML.

It just looked to me like the GP hadn't read the article and assumed it was a post by Tao. You are right about Tao of course, no objections there.

This post isn't by Tao.

The quant firms are heavy AI investors I believe.

You can also brute force the existence of a proof...

It is true that the biggest splashes have been from counterexamples and such, but I have seen many smaller examples of proving positive results in my own field.

Perhaps not superhuman yet but sufficient to completely upend the status quo. And I'm supposed to believe that this won't change in a year? or two?

Maybe, but what remains of the human professional mathematics will be unrecognizable (at least for those without tenure I guess). All of our credit assignment systems are breaking and access to computational/financial resources is becoming way more important. Math has been one of the most open academic fields but everyone is becoming afraid of sharing their ideas. Personally, my job has slowly been moving from open-ended brainstorming to prompting/digesting LLM output (or LLM output transmitted by grad students). And the students are so demoralized! Also, the academic funding structures which have supported math departments are looking less and less stable - I imagine many of them will shrink.

And getting paid…

I heard Hodge conjecture? Third-hand rumor though...

It's not impossible, but I also know of no instances of an AI being told to prove something in Lean and exploiting such tricks. Surely Anthropic also had some agents looking for issues with the generated proofs. Personally, I am also comfortable trusting Kevin Buzzard, who was leading the human team aiming to formalize FLT and discussed this a bit on his blog.

Finally, the good thing about Lean is that if anyone ever finds a new compiler bug (which, by the way, are being searched for extensively using AI), you can correct the bug and recompile any old proofs of which you are suspicious. Any tricks in a false proof must be exploiting a bug in the Lean compiler, so as we increase trust in the compiler over time we also increase trust in every previously compiled proof.

I agree it's not a 100% guarantee, but in this case the human proof is well-understood and written about by many experts, so I think Claude had plenty of material to work with. Even if the task was enormous, I don't think any of the individual steps are out of the scope of what we have seen from current AI tools.

I have never seen an AI or a human produce a false proof without explicitly using weird meta programming tricks that are very suspicious. No "good faith" Lean proofs have every been shown faulty, to the best of my knowledge. While the risk is non-zero, many of the AI companies are also trying to find bugs in the Lean kernel, so it is becoming very well stress-tested.

You just have to trust the statement and the lean compiler, not the proof. The compiler certainly still has remaining bugs, but I have never seen a bug leading to a false proof in good faith, only via obscure meta programming tricks. The nice thing is that the multiple versions of the compiler are constantly being stress tested. Still, there is plenty of work that could be done to make the compiler more trustworthy / easier to verify.

Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text