Comment by W0lf

As pointed out in my other comment, using a single image for point coloring is prone to errors due to noise, specular reflection and occlusion. I'd consider using a (normalized) cross-correlation approach with several images.