> that people develop their understanding rather than measuring against a target of a solution.
Right, but this glosses over exactly the issue that many mathematicians are upset about: you can't focus on understanding in the same way. Creative, potentially blind, exploratory work yields to proofreading machine output. One of the top comments in the thread summarizes this issue nicely: this isn't what most mathematicians want to do. I'd also argue that the change in understanding and study may, as you've pointed out, mirror chess: Magnus Carlsen has expressed exasperation with grandmasters who just memorize games and machine-computed lines rather than playing creatively. I don't think it's at all unreasonable for people to worry about similar changes in math. I find Sanderson's message polyannaish.
> I simply do not believe this is true.
I trust that many who work at these companies are decent people who, as you say, just happen to disagree with me. The people in charge are not. If we take seriously what Amodei, Altman, and the whistleblowers are saying about their work, their companies, their goals, the dangers, and the consequences of their work, then I think it's not reasonable to treat the companies has having any motive other than destructive or even deadly. The decisions issuing from on high clearly prioritize naked demonstrations of power and capability over safety and ethical, responsible behavior. It's not a Bond narrative. They're saying in essence "we're going to destroy everything"; "we're going to bring superhuman beings into existence"; "we can't control this software and it poses an existential threat to humanity"; and, in essence, "there's a prisoner's dilemma that prohibits us from acting slowly and responsibly." Even granting that "apocalyptic-millenarian suicide/death cult" is an exaggeration, I think it's a fair label.
We're in complete agreement on alignment. Your take on the hugging face incident is fascinating -- and scary. If it's true that the models misbehave because they lack adequate understanding of consequences or the damage they can do to real people, can this really be an accident? To solve the alignment problem in ways that serve the needs of industry, this kind of blindness seems like a necessity.
Thanks again for the stellar replies. You've given me a lot of think about and tempered some of my thinking. Sorry if I'm not replying with equal richness. Sick 21-month-old and no sleep.
Right, but this glosses over exactly the issue that many mathematicians are upset about: you can't focus on understanding in the same way. Creative, potentially blind, exploratory work yields to proofreading machine output. One of the top comments in the thread summarizes this issue nicely: this isn't what most mathematicians want to do. I'd also argue that the change in understanding and study may, as you've pointed out, mirror chess: Magnus Carlsen has expressed exasperation with grandmasters who just memorize games and machine-computed lines rather than playing creatively. I don't think it's at all unreasonable for people to worry about similar changes in math. I find Sanderson's message polyannaish.
> I simply do not believe this is true.
I trust that many who work at these companies are decent people who, as you say, just happen to disagree with me. The people in charge are not. If we take seriously what Amodei, Altman, and the whistleblowers are saying about their work, their companies, their goals, the dangers, and the consequences of their work, then I think it's not reasonable to treat the companies has having any motive other than destructive or even deadly. The decisions issuing from on high clearly prioritize naked demonstrations of power and capability over safety and ethical, responsible behavior. It's not a Bond narrative. They're saying in essence "we're going to destroy everything"; "we're going to bring superhuman beings into existence"; "we can't control this software and it poses an existential threat to humanity"; and, in essence, "there's a prisoner's dilemma that prohibits us from acting slowly and responsibly." Even granting that "apocalyptic-millenarian suicide/death cult" is an exaggeration, I think it's a fair label.
We're in complete agreement on alignment. Your take on the hugging face incident is fascinating -- and scary. If it's true that the models misbehave because they lack adequate understanding of consequences or the damage they can do to real people, can this really be an accident? To solve the alignment problem in ways that serve the needs of industry, this kind of blindness seems like a necessity.
Thanks again for the stellar replies. You've given me a lot of think about and tempered some of my thinking. Sorry if I'm not replying with equal richness. Sick 21-month-old and no sleep.