Intuitively I would guess that this could profit from using a more sophisticated color difference function [1] which accounts for human color perception better than simply RGB component differences. And even if one uses RGB components, making sure that gamma correction is handled properly might make a noticeable difference.
[1] https://en.wikipedia.org/wiki/Color_difference