rphv

> ...what's left?

Games, art, athletics, food, fashion, sex, social interaction. People still play chess even though computers surpassed us a while ago. We still climb mountains even though helicopters exist. Check out Ian Banks' Culture series for an interesting take on post-scarcity life with superintelligences.

> what separates it from religion?

The rational methods used to establish a large P(doom).

To definitively falsify the claim, humanity would need to live alongside superintelligence forever without it wiping us out. That doesn't mean we can't be pretty darn sure though, in the same way that we can be pretty darn sure that nuclear war could end civilization.

How cynical. What if the fears of extinction are _rational_? Given the stakes, we're obligated to treat the threat as a real possibility until proven otherwise.

People will go to great lengths to avoid believing truths that dramatically challenge their worldviews. Consider the case of Rudolf Vrba and Alfréd Wetzler, who escaped from Auschwitz and were initially met with disbelief when they described the atrocities they'd witnessed.

> If you actually thought your product will kill all humans, then you could, you know, like, just stop building the torment nexus. All by yourself, no government regulations required.

Wrong. This is a classical prisoner's dilemma. _Everyone_ has to stop, otherwise the undesirable outcome happens faster. It requires coordination, which can be achieved through regulation.

Sure, but at this point it's "save _all of us_ from my mistakes/ambition". It doesn't really matter how it started, we still need to escape the trap.

Such a bioweapon (probably) doesn't exist yet, but that doesn't mean it's impossible in principle. It's "trivial" in the sense that it's easy to _imagine_ such a thing... but a superintelligence is by definition much much smarter than humans, so whatever it comes up with will likely be very surprising to us.

Isn't it telling that Musk, Altman, Hassabis, Amodei are all signaling fear of what unchecked AI development could do? These guys can't normally agree on the color of the sky. Let's give them credit for trying to sound the alarm.

If you choose to slow down (cooperate) and the other side doesn't, you lose more than if you didn't try to slow down. In order to succeed (gain mutual advantage) both parties have to choose to cooperate without knowing beforehand what the other party will do.

Bioweapons, trivially. All it would need to do is convince one person who works in a lab, possibly under the guise of benign intent.

Or... something so far outside of the box that we'd have about as much understanding of it as a baboon does a gun.

> Navier Stokes was plagiarism

_of work that was aided/accelerated by AI!_

> So it's either they truly think AI is going to kill us all

Yes. Basically everyone here is dismissing this possibility. We shouldn’t.

Ok ok ok