Comment by ohnoesjmr

You can't really do that with distinct, as if you have 1 billion distint entries, you essentially have to store all of them to dedup.