Comment by MarkSweep

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.