Comment by erickhill

Can we not put lines in our robots.txt files to block being crawled?