rouxz
Joined 11 years ago
Karma
205
Submissions
33
Member since
Aug 14
Joined 11 years ago
"Correct" my ass
IMO the hardest things in distributed crawling at scale are a good URL frontier, priorities, rate limiting and things like that, which are quite often overlooked.