I definitely agree that consciousness is mysterious. But I think the assumption that AI is conscious begs the question "then why isn't 7zip conscious?".
Is a simple calculator conscious? How about a mechanical calculator? Is an abacus conscious? Are they only conscious when executing a sufficiently sophisticated program? What if I hypothetically had infinite time and patience, and performed the inference on paper with a pencil. Would it then be conscious?
Is a simple calculator conscious? How about a mechanical calculator? Is an abacus conscious? Are they only conscious when executing a sufficiently sophisticated program? What if I hypothetically had infinite time and patience, and performed the inference on paper with a pencil. Would it then be conscious?