Comment by nubinetwork

Just because you put something in robots.txt doesn't mean they will adhere to it.