Comment by azeemba

Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.