whaaswijk

> That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.

How does that prevent AI from becoming superhuman in all intellectual domains, thus creating ASI? We set "become good as chess" as the metric and it became superhuman in that domain.

> The problem is what happens when you max that out.

I'm not sure how you conceptualize "maxing out" the intelligence metric. Again, taking chess ability as a proxy metric for intelligence, there is no reason to believe we have maxed out chess performance, but AI is already far superior to humans. And better bots are created all the time. There is no need to design a new metric to improve chess performance. The old "How many currently existing players can I beat?" is good enough. Also, why couldn't it design better metrics after achieving superhuman intelligence?

But even if we suppose there is some kind of fixed point limit to this process, it would still be far above human level. That is all that is required for ASI.

Regarding X-risk: the point is that it becomes easy even for people who, unlike you, haven't studied biology. For things such as atomic war, the AI does not necessarily need to acquire the materials. It can access them digitally by hacking the weapons systems, possibly in collaboration with some human actors. Or maybe it spoofs detection systems causing countries to fire upon each other. Generally, it seems to me that the barrier to entry for bad actors to cause these scenarios is decreasing. Whether these scenarios are more likely than D-risk I don't know.

Maybe that's possible. But there will be a tension between how productive/useful the models can be when they are put into a very restrictive jail. In the limit they would be in a box with no way to communicate, but that wouldn't be useful to anyone. I think there will always an incentive for the people who own the models to give them more access because the increased productivity may help to outcompete their adversaries.

But let's grant that the models only communicate through certain phone lines. I think very bad scenarios are still possible. There are at least two that I can see. 1) The models exploit the users which have direct access to it. It somehow convinces them to perform tasks for it or to give it more access. 2) Control of the AI is held by a small number of people. This could be bad because it grants them an outsized power over all humans without such access, and thus lead to oligarchy/dictatorship.

I'm agnostic about whether ASI is possible. I don't have a dog in this fight other than a pro-human bias. I have used coding agents as well and I'm certainly aware that they are limited (at least in my domain).

I think the main argument for ASI is something like (i) extrapolating the progress from the past 10 years into the future, (ii) rapid progress apparently still being made, and (iii) seeing no obvious theoretical limitations.

Well yes, I'm speculating about the future. Given that, I have sketched a scenario in which a small number of people control a country or an industry, even if ASI can be controlled. If you disagree with the reasoning, I'd be interested to hear why. If not, I don't think any further discussion of my use of the term "oligarch" is going to be productive.

Thanks for the substantive reply.

> How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do. I haven't given this much thought, so maybe I'm missing something, but I don't see how this follows. One possible solution: to avoid overfitting, can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.

Regarding the X-risk vs D-risk: I think how one weighs these risks partially depends on what one thinks the capabilities of the models are. Call me a boot-licker, but if the models get smart enough to explain, in detail, to any psychopath, how to construct a bomb or synthesize a deadly virus, I don't think benefits society to distribute them widely. Therefore, to argue for widespread distribution you have to argue that either (i) the models aren't that capable or (ii) the guardrails are robust enough to prevent them from being used in catastrophic ways by bad actors. I think we may be rapidly approaching a time where neither of these hold. Having said that, I certainly agree that the D-risk is also real.

If the current path toward A(G/S)I continues, most of the technology will be owned by a small number of companies (and thus people) such as OpenAI and Anthropic. Their control could initially be economic, since they control the technologies which would conceivably replace much of human labor. It doesn't seem like a stretch to think that economic control could be extended to the political domain. Seems like an oligarchy to me.

That's true but it doesn't really address what people are concerned about. Certainly we can and should be doing a much better job currently. However, even today, with currently known capabilities, we can imagine agents breaking out of sandboxes through either known- or zero-day exploits. Now consider the seemingly rapid improvements that are being made in the field. We haven't even begun to address the potentially super-human capabilities of future models. Therefore, even if sandboxing or limiting shell access works today, it seems like we should not be confident in our ability to keep rapidly improving future models locked down.

Can you quote where in the article this is mentioned? I don't see it. I just see the assertion AI can be controlled without any substantive description of how.

So in the real world, what are the analogues to lithium and morphine we should feed to e.g. LLMS, how do we feed them, and how do we prove that it prevents unsafe behavior?

Maybe I'm wrong, but your comment seems to suggest that you are skeptical of ASI. Do you have any arguments to support that?

Btw, wouldn't AI increase the risk factors you mention such bioterrorism or nuclear war? It seems like we're not far off from AIs being able to enhance the capabilities of bad actors in the near future.

This is only one type of control, and it is certainly not infallible. Also, people will be incentivised to hook up AIs to real tools. But even if they don't, as long as people can interact with super-intelligent AIs without tool access, there are many potential dangers.

I'm not sure what you mean.

Unfortunately the article is behind a paywall. I would have been interested to see if he makes any substantial arguments. I don't see any reason to believe that AI being software entails that it can be controlled. Moreover, even if a measure of control is possible, giving that control to a handful of oligarchs seems undesirable.

What makes it so much harder to publish a CL game on Steam versus a C++ game?

Legalese seems to me a different category more akin to technical writing. I wouldn’t classify difficulty understanding a legal document as a general lack of reading comprehension. It requires specific domain knowledge.