Yes exactly. I'm intentionally using "color" as a perceptual thing, not as a physical thing. If we are talking about a color model, then it needs to model perception. As such, RGB, as a predictor of perception, can often fail because it doesn't account for much more than what hits the retina, not what happens after. For one, it lacks spatial context - placing the same RGB value with a different surround will feel different, like in the example above. But if you had a real color (as-in, perceptual) picker in Photoshop, you would get a different value.
It's excellent at compressing the visible part of the EM spectrum, however. This is what I meant by stimuli encoding.
It's excellent at compressing the visible part of the EM spectrum, however. This is what I meant by stimuli encoding.