Comment Comment by dspillett > My experience with a crawler is that people don’t actually provide a robots.txt far more often.I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures. parent dspillett 10 days ago
I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.