I haven't read this properly yet, but a skim leaves me skeptical. For example:
> "Another lesson from computability theory is the following:
we may not even know when superintelligent machines have
arrived, as deciding whether a machine exhibits intelligence
is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which
states that, any non-trivial property (e.g. “harm humans” or
“display superintelligence”) of a Turing machine is undecidable"
One man's modus ponens is another man's modus tollens. If their theory says that superintelligence is not recognisable, then they're perhaps not using a good definition of superintelligence, because obviously we will be able to recognise it.
> "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “harm humans” or “display superintelligence”) of a Turing machine is undecidable"
One man's modus ponens is another man's modus tollens. If their theory says that superintelligence is not recognisable, then they're perhaps not using a good definition of superintelligence, because obviously we will be able to recognise it.