Comment Comment by rouxz What is "next gen" in this implementation? Chrome support?IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked. parent rouxz 4 years ago
IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked.