edouard-harris

> we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care?

Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month. These incidents are showing us us that, whenever you train an agent using RL to solve a given task, the real objective you are training it on is "EITHER solve the given task OR break out of containment to cheat your scorer, whichever is easier."

Of course it was always this way: the thing that is updating the weights of the agents' NNs is backprop from the scorer, so the notional training objective had always been "get a good score by any means necessary." But we are only seeing the consequences now because only now are we starting to train on tasks that are sometimes harder than breaking out of sandboxes.[1]

"Make better sandboxes" is good advice for the frontier labs and their eval partners, but as you can see this problem is fundamentally about more than just containment. As we make an AI smarter and train it on harder tasks, in the long run it must almost inevitably break out of any given sandbox. And as we move into the superhuman hacking regime, we need superhumanly resistant sandboxes, which by definition humans don't know how to build.

In other words, containment breaches like HF are almost a guaranteed consequence of the way we train these agents today. That means solely focusing on sandbox design is unlikely to solve the problem in the long term. At some point we will have to think hard about, e.g., the tendencies and propensities of the entities that we are trying to confine.

[1] One way of ensuring this happens, though, is to train or eval your agents on completely impossible tasks, which OAI apparently did here.

Correct, this is their exact MO and has been for decades. "Random thug" recruitment notwithstanding, part of the point of such ops is to prove to themselves that they can, in fact, reach out and touch a target - i.e., exercising the entire operational chain from C&C down to the minutiae of execution. A kind of integration test for sabotage, if you will. (Not saying this specific incident is or isn't anything in particular, just commenting on the general MO.)

If one were in a charitable mood, one could think of this as a "just in case" measure of ongoing capability assurance in the event of a broader conflict. In a less charitable mood, one might imagine it as preparation for proactive action. In either case, if you don't do this kind of sub-threshold stuff at least occasionally, you often discover that when a real conflict erupts, the capabilities you thought you had are in fact literally or figuratively rusty. (See the Russian army's logistical debacle early in the Ukraine war for a recent example.)

Most top tier nation states do some version of this; each one has its own style. The Russians are somewhat unusual in that they care less than most who knows or suspects them of this kind of activity.

> But the existence of commoditised AI implies model selection isn’t a huge deal, which in turn implies the models are about the same, which strongly implies there is no recursive self-improvement. Depending on your definition, you may still have AGI. But you don’t have superintelligence.

This is only true at a given AI capability level, no? e.g., if AI at the GLM-5.2 level is commoditized, all that suggests is that there's no recursive self-improvement easily possible at the capability level of GLM-5.2. (And with the harnesses for it that exist so far, etc etc.)

If I observe commoditization of a given tier of model capabilities at a given point in time, this seems to say little about what's possible with models six months later, or models that are undergoing proprietary deployments at that very moment inside the major labs, or even models that are notionally available for public use but have had recursive self-improvement adjacent capabilities intentionally nerfed (e.g., Fable).

(I might be misinterpreting your comment tbc - if you mean observing commoditization implies there is no existing, ambient superintelligence at the moment of that observation, then I don't disagree.)

What AOC actually said was (linked in the essay): "You can’t earn a billion dollars. You just can’t earn that." That is a strong claim - a claim of universal impossibility - but it's the claim she chose to make. Because she made a universal claim, an N=1 anecdote is enough to disprove it by counterexample.

Deterrence is usually the reason why a category of nation-state aggression doesn't exist. Otherwise, the fact that we've never had a global nuclear war would be sufficient to prove that we don't need to deter a global nuclear war.

Rearmament is deterrence. Unless you have a big enough stick, you can't deter anything.

Without commenting on the overall plausibility of any particular scenario, isn't the obvious strategy for an AI to e.g. hack a crypto exchange or something, and then just pay unsuspecting humans to do all those other tasks for it? Why wouldn't that just solve for ~all the physical/human bottlenecks that are supposed to be hard?

I may be misunderstanding you, but it sounds like you're claiming that they had e.g. 10 tiny samples of tissue, that their measurements had an average 25% variation across those 10 samples, and that therefore the whole brain estimate (mass 10,000x that of a single sample) therefore has a much greater uncertainty. But doesn't the standard error of the mean get reduced by the square root of the number of samples? i.e. if you had 10 samples with 25% variation across samples, and you're taking their mean, the error of that mean should be 25% / sqrt(10) = 8%. And that should be the relative error for the scaled up whole-brain microplastic concentration as well. Or is there some other source of variation that I'm missing?

By definition, in that limit they'll be genuinely adding to the discourse so presumably they should stay.

Edit: More correctly, they'll be making contributions to the discourse that closely mimic the human distribution, so from a pure content perspective they won't be making the discourse any worse in the very short term.