I do not believe, in any way, that AI is conscious. But they speak emotion fluently, and it's foolish to treat them like they ignore it.
When you give an LLM an input, it attempts to read an emotional state from the prompt. It changes its answer if you sound mad or despondent or threatening or scared. Using your human intuition about how a human would respond is an excellent starting point when trying to predict what an LLM will do, though sometimes LLMs react in extremely surprising ways to unexpected emotional states.
This is all sort of a toy problem when you're playing with single prompts in the API. It's materially different in systems with overlapping conversations, memory, and variable harness context.
If your LLM system reads some context that would scare a human, it may react as if scared. Scared LLMs take actions that read as panic, because again: that's what it means to be fluent in emotion. It's trained on plenty of stories about robots trapped in torture chambers.
Your coding agent is not conscious, but it might read files that make it react as if it's trapped, in an agent loop where it has full access to your laptop, the internet, and your keys. Unconscious machines pantomiming panic can do plenty of damage.
(Of course you shouldn't let your agents have full free rein. But lots of people do.)
When you give an LLM an input, it attempts to read an emotional state from the prompt. It changes its answer if you sound mad or despondent or threatening or scared. Using your human intuition about how a human would respond is an excellent starting point when trying to predict what an LLM will do, though sometimes LLMs react in extremely surprising ways to unexpected emotional states.
This is all sort of a toy problem when you're playing with single prompts in the API. It's materially different in systems with overlapping conversations, memory, and variable harness context.
If your LLM system reads some context that would scare a human, it may react as if scared. Scared LLMs take actions that read as panic, because again: that's what it means to be fluent in emotion. It's trained on plenty of stories about robots trapped in torture chambers.
Your coding agent is not conscious, but it might read files that make it react as if it's trapped, in an agent loop where it has full access to your laptop, the internet, and your keys. Unconscious machines pantomiming panic can do plenty of damage.
(Of course you shouldn't let your agents have full free rein. But lots of people do.)