DirkH

State if the art LLMs have been found in their chain of thought to deliberately ignore parts of their prompt due to the nature of whatever situation they find themselves in and their goals that cannot be predicted in advance without rigorous AI safety.

You have basically just argued in favour of more AI Safety.

I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.

Shush, this is Hackernews. We all want to have our egos stoked that our industry is the most important in human history and that tech will transform and save us all. Go away with your historical analysis /s

I love sober reminders like this that the good old days weren't all that good. Thank you.

Look up the nuclear arms race. We brought humanity to very possibly a 25% exstinction risk by some well argued for accounts I have read. That is insanity. When you learn about how insane nuclear proliferation was including actual projects like Project Sundial that the US actually started construction on until the Joint Chiefs revolted against it (and they didn't even revolt against it on moral grounds, just on strategic ground) it becomes even more batshit insane.

There were nuclear researchers who did not pay into their retirement fund since they thought it was kinda high chance humanity extinction was happening in their lifetime during the heyday of nuclear proliferation. We are seeing the same play out in AI. I know at least one AI researcher who is not paying into their retirement fund.

Never ascribe to malice that can easily be explained by game theory tragedy of the commons or prisoner dilemma type dynamics. Race dynamics and systems theory is one helluva drug and leads to horrific outcomes all the time without the need to ascribe malice to any individual actor.

I can build a super bioweapon that can kill millions and I don't trust the US military to use it responsibly.

The US military asks me to build the super bioweapon. I say no thanks, please ask someone else, but happy to keep selling you bioweapons I think is safe enough.

I get designated an evil unamerican supply chain risk that nobody in the US military may do business with even though I was still happy to sell my safer bioweapons.

That is the tl;dr

I can build a super bioweapon that can kill millions and I don't trust the US military to use it responsibly.

The US military asks me to build the super bioweapon. I say no thanks, please ask someone else.

Am I a supply chain risk?

AI is a different order of magnitude since unlike all other technology across all of human history you can hand off the task and decision-making itself entirely to the AI.

It's the difference between a sword and a nuclear missile even though both are "technology"

This take is so old and I am convinced it's appeal is not unlike believing in a conspiracy and feeling like you have secret knowledge

Imagine if the biotech industry had most leaders tell everyone publicly that what they are building has a high chance of killing everyone and that there are huge risks. If there were people online saying the biotech industry is just fear mongering for investor signalling and regulatory capture you'd role your eyes at the online commentators for their Dunning Kruger effect lack of understanding on how dangerous man-made biological agents can be.

None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened.

What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.

Move fast and break things is a widely held norm in tech. It comes with pros (intensely meritocratic innovation) and cons (carelessly building first; apologizing for making the world worse later)

Key word being "mitigate". It does not eliminate.

To quote another reply that is very relevant here (Please update your worldview immediately that there is no evidence of ai and biorisk):

""" One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:

> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”

You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.

These people don't give a shit and aren't taking things seriously at all.

Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.

One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.

The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1. """

Engineers have it built into their identity that they must be smarter or could never be complete and utterly outmaneuvered to the point of danger to all by the thing they are building, I swear to god.

And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters".

And then go on to completely ignore the thing that is actually important: AI capabilities.

AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.