The problems begin with this concept of a "provably friendly architecture", which probably doesn't exist. Heck, humans aren't exactly provably friendly either.
But the main argument against this design goal of being somehow provable and intelligent at the same time is that we have fundamentally conflicting paradigms:
The first is the idea that in order to build an artificial intelligence, we'd need to build an actual artificial person. Complete with their own internal representation of the world (or as you might say: its own hopes and dreams). It's an approach to AI that is fundamentally uncertain in the way individual AI entities will turn out personality-wise, but it is guaranteed to work. We know this approach will work, because we are ourselves machines built on this principle.
==> good chance of success, somewhat limited danger
Then, conflicting with that comes this notion of a provably friendly/unfriendly design which sounds like it was made up by CS theorists who think intelligence is a function of raw processing power and thought patterns are in any way related to rigid formulas. They're very likely wrong, but luckily that also means this group of researchers will never produce anything dangerous except maybe a lot of whitepapers.
==> virtually no chance of happening, no danger
I do agree though that there might be a third kind of AI, a sort of wildly self-improving problem-solving algorithm that has no real consciousness and simply goes on an optimization rampage through the world. This would be a disaster of possibly grey goo-like proportions. BUT, this approach to AI is also very likely to be used in tools with only limited autonomy. And because the capability of an unconscious non-entity AI to understand the world is limited, the probability of it taking over the world also seems limited.
==> small probability of autonomous takeoff, but if it happens it will be the end of everything
But the main argument against this design goal of being somehow provable and intelligent at the same time is that we have fundamentally conflicting paradigms:
The first is the idea that in order to build an artificial intelligence, we'd need to build an actual artificial person. Complete with their own internal representation of the world (or as you might say: its own hopes and dreams). It's an approach to AI that is fundamentally uncertain in the way individual AI entities will turn out personality-wise, but it is guaranteed to work. We know this approach will work, because we are ourselves machines built on this principle. ==> good chance of success, somewhat limited danger
Then, conflicting with that comes this notion of a provably friendly/unfriendly design which sounds like it was made up by CS theorists who think intelligence is a function of raw processing power and thought patterns are in any way related to rigid formulas. They're very likely wrong, but luckily that also means this group of researchers will never produce anything dangerous except maybe a lot of whitepapers. ==> virtually no chance of happening, no danger
I do agree though that there might be a third kind of AI, a sort of wildly self-improving problem-solving algorithm that has no real consciousness and simply goes on an optimization rampage through the world. This would be a disaster of possibly grey goo-like proportions. BUT, this approach to AI is also very likely to be used in tools with only limited autonomy. And because the capability of an unconscious non-entity AI to understand the world is limited, the probability of it taking over the world also seems limited. ==> small probability of autonomous takeoff, but if it happens it will be the end of everything