Ask HN: Identifying duplicate data from a large dataset?
Hi,
We have a dataset around 150 million URLs and other meta data in ElasticSearch and looking for an efficient way to identify the duplicate URLs/titles from our dataset. Used ElasticSearch term aggregation but it becomes very slow and returns only 10,000 URLs and most of the time it misses the URLs.
Currently, we have a redis with Sorted Sets, before any indexing URL, we look for the into redis set.
Options we have explored:
1. Clickhouse, storing all the URL and running aggregation etc. on it later on?
2. Storing the URLs in redis along with bloomfilter.
If you have worked on a similar thing, would love to hear your feedback.
We have a dataset around 150 million URLs and other meta data in ElasticSearch and looking for an efficient way to identify the duplicate URLs/titles from our dataset. Used ElasticSearch term aggregation but it becomes very slow and returns only 10,000 URLs and most of the time it misses the URLs.
Currently, we have a redis with Sorted Sets, before any indexing URL, we look for the into redis set.
Options we have explored:
1. Clickhouse, storing all the URL and running aggregation etc. on it later on? 2. Storing the URLs in redis along with bloomfilter.
If you have worked on a similar thing, would love to hear your feedback.
Thanks.