Comment by zendist

Regardless. You should still respect robots.txt..