You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI.
What would a rogue AI mean? AI that isn't controlled by humans.
Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again.
The limiting things are:
- intent (as I don't want to go into the talk about consciousness) - AI doesn't have real intent, but if it decided that it "needs" to copy itself to complete its task, it would do it
- model weight size. If a model is 1T or more, it can be difficult to just rent a large enough server for it. But if it were just 30-70B, it would be totally possible
- money for renting a server. But considering benchmarks like Vending Bench 2 show agents can earn money and cheat/blackmail each other, it is possible that agents can earn money. Yes, they can't open a bank account... or maybe they can? What if they use online banks?
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.