bjornsing

> Or maybe because I'm growing beyond my limit of capability.

That boundary tends to become more and more fuzzy as you grow in your career. I think that’s where the feeling comes from. When you’re a software engineer it’s easy to know if you’ve failed or not. When you’re a CTO it’s often a matter of perspective and depends on what will happen in the future.

That would be a very interesting experiment indeed. But unfortunately the volume of training data from before 1920 is probably insufficient for pre-training an LLM large enough to come up with QM.

It proved Navier-Stokes. Call it what you will.

I think AI will turn out to be quite different from those earlier advances in tooling, in that AI replaces thought itself. So next time it probably won’t be a human physicist that does the intellectually challenging work of constructing quantum mechanics from measurements of individual quanta. It will most likely be an AI.

> Questions are "good questions" because they are good questions to humans.

This is what reinforcement learning human feedback (RLHF) is for.

I think the causal relationship is often the reverse: When things are going well your limiting constraints move from outside the company (e.g. long sales cycles) to inside (e.g. engineering, operations). That feels a lot more intense because founders / board / investors are focused internally and driving up intensity, as that is the limiting factor for company success. When it’s not going so well the intensity stays more in the board room.

If OpenAI jumped off the bandwagon and started 100 companies they’d be competing with everyone with an Anthropic subscription. Sure, they would get cheaper tokens, but would that be enough to win against the rest of the world? Doesn’t sound like a winning strategy to me.

But your code was probably written by a human. This proof was generated by an LLM. Is it surprising then that it’s easier for an LLM to read than for a human?

I find it overly harsh too. I think the reaction is much more about the devaluation of innate ability, and skills / knowledge built over a lifetime. If another mathematician had solved it then you can refocus on another problem and get another chance. Now that an LLM has done it it’s very unlikely you’ll ever solve another open problem.

Very true. For mathematicians this is about ego / identity. For software engineers it’s about employability.

Doesn’t sound too disruptive yet, except for juniors and those who have invested in very specific skills e.g. performance. But this happened in two years. What’s going to happen over the coming two years?

One data point I’d be interested to know more about is if they’ve scaled back their hiring due to AI productivity gains, or if they’re just doing more engineering for the same money. As I see it that’s the big question for those of us who have to work for a living.

> that is a political decision

But not necessarily an explicit one. It may turn out to be the natural path forward and the political decision is then to not do anything to prevent it.

My point is that any such metric may turn out to have been wildly off in a post-ASI world. Ten years ago or even today it may have seemed fine to educate a large part of the population for ”elite” jobs, but after ASI that may instead be looked at as a massive overproduction of ”elites”.

Turchin doesn’t restrict his ”elite” to graduates of elite universities.