Comment by xena

Is there support for robots.txt so service operators can opt out of your mass scraping?