holmesworcester

I often think about how there's an open question in this case of whether it's necessarily anti-social for a market participant to fight hard to create the level of critical mass necessary to give most people the product they want.

Everyone likes stuff showing up at their door reliably wherever they are, and this isn't possible unless you have a service with warehouses and dense customers ~everywhere, and this isn't possible unless enough people use that same service. So in one sense it's in everyone's interest to have an assurance contract where everyone says "I will use this service if you do." Then in that sense Amazon's price rule is building a mechanism to support such an assurance contract: you can't force people to sign such a contract, but you can prevent people from defecting to slightly lower prices outside of the "stuff just arrives fast" system.

In a system that was very good at collective action outside of markets, a cleaner solution in terms of consent and power might be to build an actual explicit assurance contract for an app-to-door-everything-delivered-fast service, get everyone to sign it, and make a public utility and/or separate dis-aggregated markets for the different levels of service provision, whatever they are (consumer facing app, supply, warehouse, delivery, etc.). But anyone who has ever even thought about doing anything like this will know that it's much easier said than done, and will appreciate the kludge Amazon and their investors pulled off, at least a little bit, the next time they buy a thing online.

Maybe the answer is just: use Amazon until technology (robotics e.g.) makes it easy to create something better? At this rate, we shouldn't have to wait too long?

Can't LLMs also prompt LLMs?

Are we confident that no existing LLM is capable of similarly effective prompts to those this author used? (I agree it's a stretch, but would not reject it out of hand.)

Even if not yet, will the existence of this repo soon change that, because LLMs will soon ingest it?

These details are fascinating, thank you for posting. Look at the structural problem they reveal.

1. we have a substantial societal need to lock down powerful in-training AI

2. many are calling for making these companies criminally liable for hacking

3. security experts (like yourself) are turning down offers to help secure these systems in part because of potential liability

This indicates that #2 might be the wrong response. Security professionals are like lawyers supposed to be paranoid and think worst-case. If you want top security professionals to secure these systems, we might need a culture of FAA-style blameless retro.

A stronger steelman is that it results in resources getting allocated in smarter/healthier ways across society.

If there's some business that's getting by but the land it's on is more valuable (e.g. for housing) than the business, some investors buy the business, sell the land, make the business account for the land value, wind the business down if it can't, and there are apartments there a few years later.

There's also the problem of models knowing they're likely being evaluated, even in realistic tests.

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer!

Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!

Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."

IRL paperclip maximizer: "Well first we should rob a bank..."

He was a friend of mine and it was pretty clearly suicide.

It is fair to say they hounded him with lawfare out of thoughtless careerism and provoked his suicide.

I didn't like it at the time either. My sense following the news was that it was less arbitrary than it seemed at first, but I'm against restrictions on making existing models public in general.

(It's clear now that they can do plenty of harm before they are made public.)

But it's a proof point that regulation is possible, even over the objections of the companies.

Not necessarily. The Trump administration slapping export controls on Fable, and then setting up a pre-launch review process, is a kind of regulation. A fairly aggro and controversial one, even.

If this administration actually becomes convinced that some imminent training run is likely to kill everyone, why wouldn't they act?

The key is winning the debate that ASI is species-cide by default.

We have to win it either way, because the 2028 US elections have little or nothing to do with what Xi does.

This.

Also because when encountering a new socio-technical problem it is very non-trivial to determine which one of regulations or technical solutions are easier or more effective.

To even make a good guess you need to be an expert in both domains, which is extremely rare especially in this case.

I wonder if you could take an x-risk case to court and convince a judge and jury to award damages for harm that could have happened.

Is there any precedent for this? My hunch is that it's impossible in the US at least but who knows?

Nope! :(

Meaning, people and LLMs are finding 1=0 bugs in formal verification tools. I have no idea how likely this is in this case, though!

I still routinely have this experience too. But Sol and Fable feel closer and I have this experience less with them than with their predecessors.

The people who've thought the most about this put it differently:

Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.

On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.

If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?

Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?

The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)

Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.

So by definition the only ones that appear are the ones that are not visible to monitoring.

If:

1. you have something that can find RCE's in leading commercial systems

2. its training gives it drives to communicate successfully with its peers

3. you are a leading commercial system

4. you run it ~10^10 times (the number they gave in the talk)

...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.