Comment by rvbissell

Someone who doesn't care about polluting our corpus is not going to care about your robots.txt