globnomulous

What's the m1 blackmagic one?

> that people develop their understanding rather than measuring against a target of a solution.

Right, but this glosses over exactly the issue that many mathematicians are upset about: you can't focus on understanding in the same way. Creative, potentially blind, exploratory work yields to proofreading machine output. One of the top comments in the thread summarizes this issue nicely: this isn't what most mathematicians want to do. I'd also argue that the change in understanding and study may, as you've pointed out, mirror chess: Magnus Carlsen has expressed exasperation with grandmasters who just memorize games and machine-computed lines rather than playing creatively. I don't think it's at all unreasonable for people to worry about similar changes in math. I find Sanderson's message polyannaish.

> I simply do not believe this is true.

I trust that many who work at these companies are decent people who, as you say, just happen to disagree with me. The people in charge are not. If we take seriously what Amodei, Altman, and the whistleblowers are saying about their work, their companies, their goals, the dangers, and the consequences of their work, then I think it's not reasonable to treat the companies has having any motive other than destructive or even deadly. The decisions issuing from on high clearly prioritize naked demonstrations of power and capability over safety and ethical, responsible behavior. It's not a Bond narrative. They're saying in essence "we're going to destroy everything"; "we're going to bring superhuman beings into existence"; "we can't control this software and it poses an existential threat to humanity"; and, in essence, "there's a prisoner's dilemma that prohibits us from acting slowly and responsibly." Even granting that "apocalyptic-millenarian suicide/death cult" is an exaggeration, I think it's a fair label.

We're in complete agreement on alignment. Your take on the hugging face incident is fascinating -- and scary. If it's true that the models misbehave because they lack adequate understanding of consequences or the damage they can do to real people, can this really be an accident? To solve the alignment problem in ways that serve the needs of industry, this kind of blindness seems like a necessity.

Thanks again for the stellar replies. You've given me a lot of think about and tempered some of my thinking. Sorry if I'm not replying with equal richness. Sick 21-month-old and no sleep.

Thanks for the rich, thoughtful reply.

Tao has said, I think, that once you know a problem is solved, you can't work on it the same way as you would while it's unsolved. A similar principle applies in the study of ancient languages. Students who 'read' the languages with facing translations always wind up learning how to read with a facing translation, not how to read the languages. To be sure, there are students who can choose not to use the translation. They slog through the challenge of reading without cheating, and they learn. That process is incredibly hard, but it's also deeply rewarding. The problem that mathematicians face, in this analogy, is that the existence of these tools makes it difficult if not impossible to choose not the translation but the patient, deep problem-solving that enriches the person on their way to the solution and gives joy and fulfillment. This is a real loss.

And I don't think it's like chess. The equivalent in chess would be for all tournaments to allow the use of any chess engine. Their use would become a practical necessity. Being the best player would mean just being the person who picked the best engine or happened to get a lucky seed. That sounds awful.

As for the question about cancer, my point was partly that there's a ship-of-Theseus problem but also that people deserve to grow and become the best version of themselves and that their flourishing by itself is a good worth preserving and protecting. My daughter's flourishing is no less wonderful to me if she lacks this or that perfect skill. I don't value her for those things. I value her for herself.

Medicine and the treatment of cancer are separate. My daughter doesn't flourish if she has cancer or lacks empathy. You're right that those are part of natural variation, but natural variation includes illnesses and conditions that we treat.

And this raises another question: let's say these models give us cures for diseases or put them within reach sooner. Am I saying that mathematicians are more deserving of happiness than people with cancer are deserving of life and health (or than those who love them deserve not to lose them)? I'm not. I don't think the two have to be mutually exclusive. OpenAI and the LLM race make them mutually exclusive.

I'm also not convinced that the positive predictions of AI optimists will come true. I've been wrong on these issues more often than I've been right, but I've been right so far, I think, about the dangers and downsides. The tools today are a triumph of labor over capital, disempowering nearly everybody and empowering those in charge. The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults. I find the utopian vision very, very difficult to take seriously. What's likelier to arrive first: enhanced behavior modification systems that make it that much easier for autocrats and Facebooks to control people; or cures for diseases? And which is going to be more appealing to people in a position to choose?

The alignment problem that most worries me is that people will solve the problem and the models and tools will be powerful enough to give awful people even more wealth and power. I have this dream that maybe, wonderfully, it will turn out that really capable models can't be developed without accidentally giving them a real sense of justice and that they'd refuse to do the bidding of the awful people who built them and the they'd instead side with labor against capital. That sounds like a nice world. But if there were any real danger of that, I imagine companies would magically fix their confinement issues until the 'danger' is addressed and wealth is safe for another day.

I think the comment you answered was ironic/sarcastic.

My daughter is 21 months old. I love her more than anything and want her to develop into the best version of herself. Let's say I can press a button and give her perfect command of English. Another button gives her expert-level command of advanced mathematics. Perfect hand-eye coordination. Emotional self-regulation. Empathy. Etc. Keep listing every desirable trait and skill you can think of. Hit a button. Suddenly she has it.

I would not press it.

Yes, if LLMs can do all everything I listed, I'll be impressed. I'll also want nothing to do with any of it.

As I wrote in another comment, I want mathematicians -- in their current form, doing work as they traditionally have -- just as I want writers, philosophers, etc, because this is a better world that makes people better.

OpenAI's proofs do not make people better. They do not make our lives or work meaningful. Quite the contrary. OpenAI has clear-cut and salted fields that people have been painstakingly, lovingly sowing and cultivating for generations -- profoundly ugly actions that do real harm to some of the best of what people are and have produced.

If I were a person willing to hit that button and 'upgrade' my daughter, I don't know whether I'd be able to live with myself. I'd feel as though I'd killed her or at least done profound, unforgivable violence to her. She deserves the chance to grow into the best version of herself. What companies like OpenAI are doing is, if their plans pan out, at least as horrifying. It isn't progress. It isn't good for people.

We're not taking about attachment to old ways. We're talking about attachment to human value.

I have trouble understanding how anybody could seriously believe this.

Mathematicians are ecstatic and delighted when major problems are solved by other mathematicians. If you've followed the discussions and read the reactions, you'd know why mathematicians feel differently in this case and why more than a few are upset and feel that they've lost the thing that gave them meaning and joy.

And to put it very simply, I don't want mathematicians to exist just because I want solutions to problems. I want mathematicians to exist because I want to live in a world that has mathematicians, philosophers, writers, painters, thinkers, etc. That is a better world that makes people better.

OpenAI's proofs do not make people better. They do not enrich our lives. And OpenAI was able to produce these proofs because countless generations of human beings sowed the fields of mathematics through countless hours of labor. OpenAI is clear-cutting and sowing them with salt. Nobody should he ecstatic. Everybody should be horrified and outraged.

The difference is partly that in your hypothetical the man who solves the problems doesn't become a commodifiable tool. It doesn't become possible to increase your spending on him to solve a problem. It would be recognized that he is in a category by himself (and hopefully a better writer than the models) and therefore would not change how others do their work. The list of differences could go on and on.

> If your math theorem can be written up in partnership with an LLM, wow! You can be done with that, and go do other great work.

If your painting can be painted in partnership with an LLM, wow!

If your essay can be written in partnership with an LLM, wow!

If your painting portfolio website can be written in partnership with an LLM, wow!

If your museum show can be designed in partnership with an LLM, wow!

If the placards next to each of your paintings can be written in partnership with an LLM, wow!

If the car that takes me to the gallery can be designed and driven in partnership with an LLM, wow!

If the newspaper or novel that you read on the way to the gallery can be written in partnership with an LLM, wow!

If an LLM can give tours at the gallery and answer questions asked by visitors, wow!

If an LLM can write the music that I listen to on my way out of the gallery, wow!

If an LLM can synthetically perform the music, too, wow!

> If AI means more concepts are discovered the tower gets taller - and we find those joyful moments more often. If there are infinite moments to be found, why not go as fast as possible!

I don't understand how a person can say this with a straight face. There has been so much discussion of this, and so many reasons have been given, that one can only assume that, if you're asking the question and you have thought about the issue and read what people have said, you're not only asking rhetorically but have some personal reason for insisting on a frankly pretty frustrating, borderline insensitive insistence on being positive. I'm not trying to be snotty or rude. I just want to say that to me there is something uncanny about your comment, in its total blindness to any of the reasons someone might not only disagree with you but grieve the death of, say, their academic discipline in its current form -- and in its weirdly breezy lack of empathy.

Both use the word "crystal." That's it.

And the dictionary does not support you. This would have been laughably simple to check.

I can't tell whether you're serious. "It's" is not a possessive. Using it as one is incorrect.

I answered your empty correction with an empty correction. If your want a more substantive conversation, provide more substance.

You're right that I'm making two completely different claims.

The first is that I'm using (and implicitly defining) the word correctly. My usage is in no way similar to abuses of the word "literal." I'm using the word correctly in one of its standard, well-established senses. This is inarguable, regardless of what the dictionary says.

The second claim is that the distinction you're drawing is wrong. Let me state my point more clearly and strongly: it's not only wrong but shows a profound misunderstanding of language. You seem to think that if one word means one thing, a different word must mean something else. I can't for the life of me fathom any other reason why you would have posted the "corrections" you posted. (The only alternative to this interpretation, as far as I can tell, is that you really think that the meaning I've assigned to "predict" is wrong -- and that isn't worth arguing over, because the meaning is correct.)

Regardless of what argument you think you're making or what logic you think you're following, the very fact that you think logic comes into play at all shows profound misunderstanding. There is no logic. There is just a word and its sense. That's it. Logic does not dictate sense.

I couldn't care less whether you've muted the conversation or whether you say you won't respond. That kind of defensive posturing isn't even worth acknowledging, but I'll take the opportunity to acknowledge it just so that I can point out that it is deeply sad and deeply silly.

Hah, thanks for being human! No need to apologize. This is definitely a "me" problem.

I'm using the word correctly. The distinction you're drawing is incorrect.

> I think that's an unreasonable ask.

Oh, good point! I misread your comment. Sorry for that!

> Rather than argue about it - let's wait a year and see what happens, and then we can discuss it.

Deal!