Since the order of the samples doesn't matter, could you sort them somehow so the Gzipped stream of samples can be compressed better? (E.g. sort the color index by component average, by red, by maximum component, ... and sort the point index by color.)
Have you tried struct-of-array (AAA...BBB...) instead of array-of-struct (ABABAB...) layouts?
Have you tried struct-of-array (AAA...BBB...) instead of array-of-struct (ABABAB...) layouts?