_alternator_

This is a very nice discussion, but it bothers me that what is clearly a plain error in their ZoomIn paper is being treated a bit cavalierly. (The error was claiming a neuron was a "dog and car" neuron, when in fact it was a V neuron. ZoomIn wasn't the first to make the error, but it definitely promoted it.)

This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude.

I suppose there are different standards in different scholarly communities. And maybe it truly was impossible to know at the time that the dog/car relation was actually a V shape, but my recollection is that this type of feature was well known at the time and so the error could have (should have?) been caught a decade ago.

So: massive new discovery, or a spin on a correction that's lead scholarly research astray for almost a decade?

Yes, but writing down the address in the lookup table takes log(2^n)=n bits, so the claim of an O(1) lower bound is simply wrong.

They also use this result to great hindbrain neurons in the lab for the first time, so it's a strikingly useful discovery.

I think it's also fair to say they are different organs based on the developmental physiology. Many other organs come from distinct embryonic cell lines.

I really think the right way to look at this is as a matter of degree ('what is it conscious of) rather than whether it is conscious or not. The crux: if the capacity exists in all material but "something else" makes it conscious, then consciousness is outside of the material, right?

The argument that consciousness suddenly arises due to a particular configuration of atoms is also suspect; all forms of consciousness we routinely accept are fairly robust to many configuration changes. It seems to me that consciousness hinges on the discussion of degree or content, not on the material form. A soup of atoms certainly could be aware, but not aware of in the same way that "I" am aware.

Materialism implies Panpsychism. Not a surprise, but this is put forward in a confrontational way.

The real question for Panpsychism is not whether something is or is not conscious, but conscious of what. If everything is aware, what is this thing we have that a rock doesn't seem to have (but LLMs might have)? I'm basically sold on the premise of Panpsychism, but we do have to recognize that when we say "consciousness", we aren't always talking about the same thing.

I'll give you the benefit of the doubt. GP was referring to the use of copyrighted material to train LLMs.

I think "close to completion" is not the right framing. Creating good open problems was an achievement because these problems often sit at the edge of known techniques, and solutions require inventing "new math". It's hard to find these problems, and they take decades to mature as they withstand scrutiny by many people.

In another comment below, I likened this to clear-cutting a forest. Growing the forest takes a lifetime; destroying it could happen in the next few months.

I mean, the oracle doesn't really seem so hypothetical right now. And clearly it's going to drastically change these fields, and mathematics, particularly pure mathematics, must change most of all in order to adapt to the existance of a math oracle (or something close to it).

Yes, and they will. But what's happening here is that the system that cultivates mathematics (and mathematicians) is recieving likely the biggest shock of its history. How do you reward merit and identify talen when people can't absorb the number of proofs being generated, much less understand them? Perleman's proof of the Poincare conjecture took several years for the mathematical community to digest; the proof of Navier-Stokes will probably take a similarly long time. In the mean time, it looks like all open problems will be solved (or proved that they can't be solved).

It's not that the horizon is expanding because of this. It's more like a forest getting clear-cut.

This series of posts by Terry Tao is a direct response to the Navier-Stokes results (multiple results!) from the last 24 hours. The question is what is left after the levelling of mathematics, in all its senses, occurs? How can you protect a field that's under this much pressure in the next 6 months?

> [I]t is now the identification of a promising problem which is the scarce and precious resource. We have now seen that even the rumor of someone working on a problem can trigger a massive amount of AI-powered effort to flatten it before the original research project has time to reach its full potential.

> The significance of this with respect to the way we train students, assign credit, referee, and decide what is worth one human life’s attention cannot be understated.

This. What is worth a human life's attention? As little as a month ago, mathematics was valuable in part because only a small number of people could possibly make progress on the frontier. We are confronting an existential moment for a 4000+ year-old human cultural endeavor. The assumption that "mathematical thinking is hard" has been built-in at a number of important points in how we support mathematics and mathematicians.

We need a different model, and fast. Already, the research community is feeling unable to digest proofs fast enough to keep up with the output of AI models. The paper is 165 pages, and the discovery was finalized two days ago. What this means is that nobody really understands it. Nobody would accept OpenAI's proof in this amount of time, except that they formalized it in lean. The formalization alone would normally be another years-long (or career-long!) effort if the world was the way it was one year ago.

So, again, what efforts are worth a life's attention today? It's a harrowing change.

The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.

That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.

The claim seems speculative, but grounded. For example, as part of the hugging face attack, the agents began signing messages because they were worried about impersonation on a publicly accessible message board. It's only a small step to use public key encryption between agents. Once you are posting public keys, a private messaging is readily available.