Comment by jeremyliew

I don't understand why you'd want to stop crawlers. If you didn't want people to see your content, it probably shouldn't be on the public web.