Comment by cordite

The first thought I had would be to put this into HSV, and then index that with an R-tree [1]. This way, you can do nearest neighbor kind of stuff, similar to geospatial indexing.

[1]: http://en.wikipedia.org/wiki/R-tree