resident423

You mean for AI consciousness or existential risk? I think Anthropic downplay the existential risk (they might talk about it now and then, but they still build frontier models) but are overly confident about AI being conscious, and I think these two things are pretty strongly related to each other.

I agree with this, but I don't think Anthropic would, they appear to be much more convinced than what I think would be reasonable.

It seems like a simple solution but if Anthropic was actually to dedicate all its resources this way wouldn't the investors just demand a new CEO?

I think Anthropic believe these risks, but I also think they've spent so much time talking to Claude that they've pretty much lost their minds now. Anthropic have a model welfare department and have numerous times suggested that Claude is conscious and has human like emotions.

My smoke detector has gone off three times now, where is the fire?

I suppose you end up hating it either way, if you're a better developer than it you don't like it because it sucks, and if it's a better developer than you then you just feel obsolete

You mean that self improving AI is very far away? I'm not sure how much I believe that now it's solving Erdos problems?

> but can't come up with revolutionary theories that were never in the training set.

Can you elaborate? I don't think the solution to the unit distance problem was in the training set, but I'm guessing you mean there's some higher bar for revolutionary theories LLMs cant reach? If so where do you expect the limit will be?

I agree with all of these points, my view is just that open source doesn't really do much to prevent it. I also think it adds the additional danger of making dangerous capabilities widely available to anyone, like the ability to design novel viruses which is something that we can't really defend against once it's out there. If anything, putting this kind of capability in the hands of anyone with a GPU could create justification for a mass surveillance state or further concentration of power.

I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.

I can understand this for hacking tools, but I'm not really sure how we fix the security issues on the CBRN side? We can't patch the human body like we can with software, so if the model has strong biological capabilities and is released open source, what stops it being used to construct new viruses and things like this?

I would say I agree with Anthropic on open source for the reasons stated above like cyber crime, CBRN etc, but I'm interested to hear the other side of the argument. What would be the argument for open source over closed source?

At the moment is the significant part, how long do we expect this to last now that AI is capable of generating novel ideas like solutions to Erdos conjectures?

>CONFLICTS OF INTEREST

DC has conducted industry-sponsored research involving creatine supplementation and received creatine donations for scientific studies and travel support and speaking honoraria for presentations involving creatine supplementation at scientific conferences and on social media. In addition, DC serves on the Scientific Advisory Board for Alzchem and Create (companies that manufacture creatine products) and as an expert witness/consultant in legal cases involving creatine supplementation. NF declares no conflicts of interest

A nice unbiased study, I may as well ask Exon Mobil for their opinion on climate change at this point

I don't think that's true, from what I've seen the people on here are pretty consistently anti surveillance

All of these discussions seem to assume there's a limit at human level intelligence beyond which AI can progress no further. What stops the AI companies taking their human level model and training it even more until it's superhuman?

Pdoom would be the most important for me, everything else depends on us being able to control the AI.

But beyond that there's still problems like concentration of power and surveillance, permanent loss of jobs, cyber and bio security. I'm not convinced things will go well even if we can avoid these problems though. I try to think about what the world will be like if AI becomes more creative than us, what happens if it can produce the best song or movie ever made with a prompt, do people get lost in AI addiction? We sort of see that with social media already, and it's only optimizing the content delivery, what happens when algorithms can optimize the content itself?