Currently, using ElasticSearch for storing the meta data and other raw data information but it is a very small scale around 500,000 domains.
I have been tasked to scale it to 20-40 million domains and storing their internal/external links while building a page rank/domain authority score for each domain which we are adding to our database.
What do you guys suggest/recommend for storing this data at a very large scale as web page internal links/external links will be stored which will lead it over 100M-1B links database?
Any kind of feedback/suggestion would be appreciated.
Currently, using ElasticSearch for storing the meta data and other raw data information but it is a very small scale around 500,000 domains.
I have been tasked to scale it to 20-40 million domains and storing their internal/external links while building a page rank/domain authority score for each domain which we are adding to our database.
What do you guys suggest/recommend for storing this data at a very large scale as web page internal links/external links will be stored which will lead it over 100M-1B links database?
Any kind of feedback/suggestion would be appreciated.
Thanks.