Comment by latenightcoding

when I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.