unsui

It would appear that the presiding heuristic is:

"Do whatever you can get away with, with prioritization on local optimization with eager evaluation"

So basically, be an asshole/sociopath. The tennets of objectivism.

"morality" is basically memoized reinforcement learning over many human generations. So, it seems that the cache has been wiped out, and they need to learn again why they shouldn't be focusing on local optimization exclusively.

maybe that product doesn't deserve to survive.

"Hmmm, but if I cap the Torment Nexus, the competing Torment Core will win out, we can't have that!"

Agreed.

The real alignment issue isn't in the challenges getting AI to align to human values.

It's in humans understanding what is worth aligning to.

"He chose... poorly."

"sarcasm" is a cop-out.

The writer can simply say "it was a joke!", cuz we all know someone who uses this pattern regularly when they are trying to squeeze in a non-popular but highly lucrative (to the messenger) position to test the waters.

Either it lands (in which case, it was always meant to be taken literally), or it doesn't land (and can always be walked back as "it was meant sarcastically" or "can't you take a joke?"

The cost proposition to sending this message payload is low, since the messenger can always wiggle their way out (therefore, a "cop-out").

Therefore the need to be more explicit (i.e., /s) in a medium with little nuance affordances for tone and the significantly-increasing risk of such non-popular messages being taken literally (particularly by the also significantly-increasing audience of robots, who themselves may not have the capacity for nuance or irony detection).

Tl;Dr

It's a cop-out, and should be stopped.

The fact that, on HN, you can't be sure of that, tells you all you need to know.

sadly, Poe's law scales exponentially on HN

No, this is not good, in any way, except for AI shareholders (of which I assume you are one, else not rational to support this position).

Paperclipping here, greedily optimizing for shareholder value above all else. Please stop.

Gonna put out a blanket assertion about my preferences, to get a read on whether these are shared or not:

As humans, we have directives (genetic, cultural, societal, etc.) to prioritize humanistic endeavors (and output) above all else.

History has shown that humans are overwhelmingly chauvinistic in regards to their relationship to other animals in the animal kingdom, even to the point of structuring our moral/ethical/legal systems to prioritize human wellbeing over that of other animals (however correct/ethical that may ultimately be, e.g., given recent findings in animal cognition, such as recent attempts to outlaw boiling lobsters alive as per culinary tradition).

But, it seems that some parties/actors are willing (i.e., benefiting) from subverting this long-standing convention (of prioritizing human interests) in the face of AI (even to the point of the now-farcical quote by Sam Altman that humans take far more nurturing than LLMs...)

So: should we be neglecting our historical and genetic directives, to instead prioritize AI over human interests? Or should we be unashamedly anthropic (pun intended), even at the cost of creating arbitrary barriers (i.e., the equivalent of guilds) intended to protect human interests over those of AI actors?

I strongly recommend the latter, particularly if the disruptions to human-centric conventions/culture/output are indeed as significant (and catastrophic) as they will likely be if unchecked.

Thank you for your new substack, very illuminating (and scary) reading.

It's as if someone was watching the show "Person of Interest" as a guidebook for how to build (and weaponize) the all-seeing eye of sauron that we have today.

> Oracle will be the first to freeze

one can hope

the whataboutism is strong in this one

There are a lot of things that are "perfectly legal" to do. Doesn't mean that, in practice, law enforcement would necessarily take the most extreme legal action possible.

This is a new development (well, new-ish for many communities... I imagine predominatly-black communities have always experience this) whereby LE is explicitly instructed to look for any legal course of action to punitively enforce the law (rather than using a more judicious interpretation, which was more in line with the spirit of many of these regulations).

So yes, technically, law-abiding citizens should always xxxx. Does that mean that, in real life, folks always do this? Only if they are in a paranoid state whereby LE maliciously enforces the law for any minor violation and enforces overwhelming (often illegal) responses to these infractions.

>Our PMs don't like making things opt-in

That is an implementation detail. What matters is the outcome:

Notion leadership has signed off on this being opt-out.

The calculus here, as you indicated, was that opt-in has little buy-in.

What leadership didn't take into account was the risk of this being publicized, and the blowback from this awareness.

That, or leadership has already calculated that not enough people will care (possibly true).

I suppose it's then up to those that do care to make more noise about this, to tilt the odds?, so this specific calculus (also known as enshittification) doesn't keep occuring (i.e, if the blowback costs are disproportionate to the value provided by default opt-out....)

Subnautica in VR (with the Submersed VR mod) is easily the most immersive environment I have ever encountered.

not entirely.

The risk raised in the article is that AI is being promoted beyond its scope (pattern recognition/creation) to legal/moral choice determination.

The techo-optimists will claim that legal/moral choices may be nothing more than the sum of various pattern-recognition mechanisms...

My take on the article is that this is missing a deep point: AI cannot have a human-centered morality/legality because it can never be human. It can only ever amplify the existing biases in its training environments.

By decoupling the gears of moral choice from human interaction, whether by choice or by inertia, humanity is being removed from the mechanisms that amplify moral and legal action (or, in some perverse cases, amplify the biases intentionally)