> becomes "I think that I am a single agent, therefor I am such"
No, it's "Most of the time I act as a single agent, therefore my perception of myself as a single agent is not an illusion, but a simplified model". It doesn't matter how many neurons are voted, if I'm doing what I was thinking I'm going to do. Experiments can poke at edge-cases where self-model is incorrect, but that's expected.
In a case of a brain, my self-model also runs on brain's neurons and is a part of what brain is doing. It makes it even less illusionary, than an internal model of the world.
No, it's "Most of the time I act as a single agent, therefore my perception of myself as a single agent is not an illusion, but a simplified model". It doesn't matter how many neurons are voted, if I'm doing what I was thinking I'm going to do. Experiments can poke at edge-cases where self-model is incorrect, but that's expected.
In a case of a brain, my self-model also runs on brain's neurons and is a part of what brain is doing. It makes it even less illusionary, than an internal model of the world.