erikhoel

I worked on that in graduate school :)

Ah, but what about the subsections of the subsections? And so, ad infinitum? Eventually you'll need something like the exclusion postulate of Integrated Information Theory

It's really not, since we don't have anything close to a strong (or "general") AI. GPT-3 is the closest, but even it is just in that direction. So it is quite early. Good time to have the conversation.

There are all sorts of ways to build intelligences. Humans are unique in that they are mammals (defined by having mothers). Mothers raise us with love, and teach us, for our helpless first years. We also have to act in communities. So there is a sense in that we are very lucky - in humans, our intelligence correlates with our altruism. In the grand space of possible minds, it is very unlikely that altruism and morality is correlated with intelligence. So whatever that machine race we birth is, it won't have any of the things we value if we're just building for "precise perception, reasoning, etc"

Agreed, these are both good replies to that (which is probably the most common response for some reason)

I actually think most working scientists agree about The Science Game. Almost everyone thinks of things this way in their darker moments, or talks among other scientists like that, although it is a bit of an "open secret." For gain-of-function research in particular, I don't imagine it will survive this news cycle. It has been banned for periods of time before (I believe last ban ended in 2017) and there's almost no way it doesn't get and likely stay banned, at least in the USA.

Author here. Happy to be listening in on some of the thoughts of HN. If you have a specific question or thought feel free to AMA.