somesortofthing

The existence of status competition doesn't seem like much of an obstacle to this outcome. People at the bottom of the status hierarchy will drop out to live in simulations that give them what they wanted but couldn't get from life. After a while, they'd likely choose to suppress their memories of ever having existed outside the simulation - there's nobody in the experience machine that'll judge you for doing it. With those people gone, the people immediately above them now occupy that bottom level, and the cycle repeats.

Whoever is in charge could try to drag those people back into the real world by force, but it's hard to imagine GPT 12.2 Polaris or Claude 9 Cinematic Universe giving that the green light. The Status Game will end with a single, final winner.

People at the bottom of the status hierarchy will drop out to live in simulations that give them what they wanted but couldn't get from life. After a while, they'd likely choose to suppress their memories of ever having existed outside the simulation - there's nobody in the experience machine that'll judge you for doing it. With those people gone, the people immediately above them now occupy that bottom level, and the cycle repeats.

Whoever is in charge could try to drag those people back into the real world by force, but it's hard to imagine GPT 12.2 Polaris or Claude 9 Cinematic Universe giving that the green light. The Status Game will end with a single, final winner.

It can't consistently produce new, interesting-to-humans conjectures faster than it solves open problems, no. AI-generated natural-language proofs are more or less human-unreadable and are solely concerned with the process of constructing whatever object serves as the counterexample. Natural-language counterparts of formally verified proofs are often unfaithful to the formal component. All of this combines to create less-than-fertile ground for future study.

Strip mining interesting conjectures. AI can answer interesting(pedagogically valuable, carrying important implications, etc.) mathematical questions in formally correct gibberish far faster than mathematicians can generate them. Until AI can replace the conjectures it resolves on its own, it's consuming a finite resource.

It's very funny that despite the initial shock of how much models trained on next-token-prediction(plus instruct-tuning and some light RLHF) alone were capable of despite no built-in objective, every advance since has made them look more and more like the paperclip maximizers of yesteryear.

I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.

You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much.

To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.

What happens when the thing they have to be responsible for takes multiple lifetimes to understand on even the most basic level, let alone know deeply enough to feel comfortable taking responsibility for it?

Microservices can trample on each other just as easily as monolith internals can. If anything, the friction of reconciling changes in a monolith is useful signal that conflicting changes happened, and it takes slow and flaky e2e tests to replicate in microservices. It's not like you're resolving the merge conflicts by hand.

Opus 4.7+ and 5 being annoying and incoherent reads like a portent of things to come: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to spend R&D time on making the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.

Anything generally useful/novel probably does end up in some kind of central catalogue(or maybe just the weights). After a while though, you're tall enough yourself that the giant's shoulders don't make much of a difference.

I doubt many pure mathematicians would be okay with totally subordinating all exploratory mathematical investigation to specific practical problems, and the field would look very different both in terms of how it's organized and what output it produces were it run that way.

I don't see a reason this dynamic couldn't continue with zero dedicated effort toward "pure math" as such. Fundamentally, the function of math in practical application is producing deterministic(or stochastic, but obviously not in the LLM sense) models that, given initial observations/conditions, are capable of predicting some aspect of the future with reasonable accuracy. In the same way that human society does, a machine would grab the physics rung before the theory rung every time, but why does that matter? If progress is stuck, the machine turns the gauge away from "exploit" and toward "explore" and put together a bag of new tools it can sequentially try on the real-world problem.

I still don't see a role for humans in this process. They might direct the practical/physical aspect(if AI turns out to be less superhuman there) but they'd likely turn the hard conceptual problems over to the machine and never look inside the box - no human alive could understand even the smallest part of what's going on in there in less than a thousand lifetimes anyway.

Taken on its premises at least, I think this piece inadvertently fails to make a case for why "pure mathematics" should persist as a field of human or machine activity. A real-world system with the abilities of a robustly superintelligent mathematician can, when some practical problem requires it, formulate a problem statement, churn for a bit, spit out a formalized answer, and continue whatever outer-loop task it was doing without a human even finding out.

If you have a system that gives you arbitrary on-demand math results, dedicating any resources, be they human labor or compute, to producing them for their own sake just seems like a waste. Why catalog the Library of Babel?

I think Opus 4.7+ being annoying is actually indicative of something else: Anthropic is clearly all-in on building persistent end-to-end agents that act autonomously and direct agent swarms. As such, they feel less of a need to make the outputs pleasant to read for humans(especially at the cost of capability anywhere else) when humans aren't part of the intended operating environment.