Comment by akerl_

If I'm worried about badly behaved crawlers... what's robots.txt doing for me?