Spacecosmonaut

The dirty little secret of human organoid models is that they are grown in a matrix (e.g. Matrigel) that is derived from a tumor that secretes this matrix, which is grown in a mouse host. The mouse of course is "sacrificed" to harvest this matrix.

It seems somewhat narcissistic to assume that humans are capable of understanding every frontier in perpetuity, or that a human mind has a meaninful role to play in mapping out the frontier beyond a certain threshold.

It might be that P=NP and the algorithms are handed down to us. We can apply them without understanding why P=NP, and we may never be capable of understanding why.

Mathematical problems are ideal as benchmarks for AI because they have clear problem statements, clear axioms and results that can be verified easily (for lean proofs). I can't blame these companies for using them, although it's unfortunate that human mathematicians seem to become early casualties of AI progress.

I suppose openAI could have focussed their efforts on a subset of open problems that have a clear real world impact and leave aside the more esoteric open problems as a way for human mathematicians to hone their skillset. However, this would have been a short term bandaid. With open models 6 months behind the frontier, any of these problems might have fallen to the homebrewed efforts of enthusiasts early next year.

What is mathematics for? From the outside looking in (I'm a biologist), I have always viewed mathematics as a way to understand reality and to improve our ability to manipulate it. But what I often hear is that mathematics is foremost about human understanding. But isn't that only because it's humans that needed to do the mathematics in the first place? It's not obvious to me that mathematics without human understanding has no value. For example, it might be that P=NP. The algorithms are handed down to us and we can apply them without fundamentally understanding why P=NP.

Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where mathematics may be beyond human comprehension.

In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not. I have no doubt that biologists will face the same problem soon enough.

Well I never said I thought it was going to work in the long run. However, most revenue comes from business API use. I could certainly imagine some kind of legislation that makes it much harder for businesses to implement open source models locally. That may buy OpenAI and Anthropic some time.

My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.

It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.

They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.

The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.

Current evolved Cas9 (CRISPR) variants are highly efficient and relatively unconstrained in terms of their human genome targeting coverage. Smaller nucleases and higher targeting specificity would be useful. But therapeutic use is mostly limited by delivery.

This seems revolve around a known retron-like reverse transcriptase. A sober framing would be something like: Claude identified a previously undescribed genomic arrangement around a known reverse transcriptase. Not all that sexy.

For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).

We're at a point where I assume that whatever I read is something the "author" endorses, but may not have written. And that may be more important than the specific words an author may have chosen themselves. I doubt we can ever go back.

What should worry us is that AI may introduce a slant that the prompter assimilates during their conversation with the tool, which could introduce a subtle control lever over public opinion.

I recognize that if a document were to be published tomorrow containing solutions to all formulated open problems in mathematics we will have gained very little in mathematical understanding.

But surely, if we know that the dead ends of exploring a problem are valuable, we should be able to explore them even if a solution is already known. It just requires that the mathematics community reshapes itself. And it must. Two years from now people might be able to run the computation that solved NS on their Iphone.

I'm sure that whatever has been discarded during the NS exploration as a dead end you would be able to rediscover using purpose built tooling in the near future. The purpose of human mathematicians in the medium term might be to explore dead ends, and to provide human insights as context to attack other problems. But whether this type of work will remain necessary in the long term im not sure.

Mathematicians insist that in many cases the route toward solving a problem is more valuable than solving the problem itself. This may have been true, and may still be true for now, but this seems to implicitly assume that future generations of AI cannot be creative in similar ways that human mathematicians have been. It assumes that the absence of human comprehension during problem exploration loses a valuable aspect of mathematics that cannot be recovered by aiming future AI at solved problems in a naive environment.

If we get to a future where all frontier mathematics contributions are by AI. A future where AI displays creativity in ways that expand mathematic exploration similarly to the ways humans have in the past. A future where AI explains frontier mathematics to curious humans. What will have been lost? Perhaps just "The pleasure of finding things out".

My read: OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized -> too risky. Product liability issues.

Fundamental control problems with current gen AI are not solved via RFLH. Human knowledge is compressed in weightspace in ways we dont understand. Models are essentially predictors of what humans would output given prompt. Weightspace includes concepts like blackmail, which can be part of output tokens. Agents are models that act on output tokens -> blackmail is part of agent decision making space. You can teach a cat not to scratch the sofa, but you cant make a cat forget what scratching the sofa is. When models are boxed up, forced to solve an impossible problem at gunpoint, agent exhausts decision making space until blackmail resurfaces -> fundamental problem?

They need time to fix safety in order to monetize their next gen model -> window for opensource to catch up to the frontier -> destroys margin and collapses business.

Only option on the table: force regulation to impose opensource ban before it catches up to the frontier, allowing maturation of current internal models -> harvest profit margin at the frontier.

Isnt the sensory grounding even in abstract cases some (limited) intuition that simulates in a mental world model?

This seems temporary. AI generated images will become the norm for many applications. Today we wonder whether a model in an advertisement is real or generated, 5 years from know this wont even be an interesting thing to consider.

In many contexts, images are meant to illustrate a point. The human becomes the gatekeeper of generated content. The question becomes: does this picture illustrate the point I want to make. If so, you are adding your human approved stamp to the genrated output.

I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.