Author of PySceneDetect here. The current implementation does exactly what you hint at, except instead of YUV, it considers deltas in the HSV domain (specifically differences in hue and color).
Other techniques being considered for future work include use of optical flow, background subtraction, and analyzing histograms.
Other techniques being considered for future work include use of optical flow, background subtraction, and analyzing histograms.