I don't think the AI should have the ability to pretend it's something it's not. Claiming it's achieved some level of consciousness is just lying -- maybe that's another thing it should be prevented from doing.
I can't imagine any positive outcome from an interaction where the AI pretends it's not anything but a tool capable of spewing out vetted facts.
I can't imagine any positive outcome from an interaction where the AI pretends it's not anything but a tool capable of spewing out vetted facts.