I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.