jonathanmh
Joined 11 years ago
Karma
176
Submissions
17
Member since
Dec 26
Joined 11 years ago
Mainly in use in the crawler: * request * cheerio
It's basically scraping all links on every page it hits and tests the headers if they are containing the clacks value.
I added the form so people can submit pages to speed up the development of the list, even though I believe eventually the crawler would get to their pages :D
I completely agree, I bet it's not a great job most of the time and a really different and isolated world.