Comment by catlifeonmars

I don’t disagree with you about robots.txt… however, what _is_ the right tool for the job?