You're exactly right. nrc100's comment is meant to have you ask yourself a loaded question.
The implication of "don't fool yourself" is that future software of this kind, in all likelihood, will constitute a thinking, feeling mind. Moreover, there is a weaker implication that the computer will have a sort of inherent moral worth — an assumption that I think ultimately rests on an extremely shaky foundation about how moral worth should be construed (but one which tends to appeal to the biases of technologists).
I'll have to write something about this, because I don't have the time and space to get into why exactly this is, but I'm bookmarking this now.
The implication of "don't fool yourself" is that future software of this kind, in all likelihood, will constitute a thinking, feeling mind. Moreover, there is a weaker implication that the computer will have a sort of inherent moral worth — an assumption that I think ultimately rests on an extremely shaky foundation about how moral worth should be construed (but one which tends to appeal to the biases of technologists).
I'll have to write something about this, because I don't have the time and space to get into why exactly this is, but I'm bookmarking this now.