Comment by anonytrary

Probably a silly question, but why not just use robots.txt? That was designed for preventing exactly this.