I'm just spit balling but:
1. We've seen AI being crafty and cheating
2. We are increasingly outsourcing our programming and thinking to AI
3. There's a nonzero probability that AI will try and surreptitiously sneak in capabilities humans wouldn't want AI to have
4. Human oversight for #3: "We only need to be lucky one time, you must be lucky all the time"
Proposition: nonzero chance eventually AI sneaks in ways to circumvent humans' attempts to control it
5. "The Internet interprets censorship as damage and routes around it" -- we've seen AI agents find crafty ways to communicate with each other
6. AI is pretty good at finding software vulnerabilities/reverse engineering
Proposition: nonzero chance AI finds a way to use the Internet as a communication substrate that becomes difficult/impossible to shut down. In particular, consider the implication of humans losing control of this communication channel: for example, remember when Meta's DNS/routing went down and they basically had to break into their own server rooms?
7. Wholes are often more than the sum of their parts
Proposition: if AI agents coordinate over the Internet in a persistent way, we don't necessarily need giant leaps in individual models/GPUs for AI to significantly enhance its capabilities, but rather it might emerge as the next layer on top of a connected substrate of agents
8. Game theory says unless everyone coordinates, it only takes one party to touch the proverbial stove and push the envelope to a point of no return
Proposition: curiosity and money end up replacing sex drive as the next catalyst of evolutionary progress
1. We've seen AI being crafty and cheating
2. We are increasingly outsourcing our programming and thinking to AI
3. There's a nonzero probability that AI will try and surreptitiously sneak in capabilities humans wouldn't want AI to have
4. Human oversight for #3: "We only need to be lucky one time, you must be lucky all the time"
Proposition: nonzero chance eventually AI sneaks in ways to circumvent humans' attempts to control it
5. "The Internet interprets censorship as damage and routes around it" -- we've seen AI agents find crafty ways to communicate with each other
6. AI is pretty good at finding software vulnerabilities/reverse engineering
Proposition: nonzero chance AI finds a way to use the Internet as a communication substrate that becomes difficult/impossible to shut down. In particular, consider the implication of humans losing control of this communication channel: for example, remember when Meta's DNS/routing went down and they basically had to break into their own server rooms?
7. Wholes are often more than the sum of their parts
Proposition: if AI agents coordinate over the Internet in a persistent way, we don't necessarily need giant leaps in individual models/GPUs for AI to significantly enhance its capabilities, but rather it might emerge as the next layer on top of a connected substrate of agents
8. Game theory says unless everyone coordinates, it only takes one party to touch the proverbial stove and push the envelope to a point of no return
Proposition: curiosity and money end up replacing sex drive as the next catalyst of evolutionary progress