robots.txt is not an actual Internet standard, and there are no standard controls for when a bot ignores its contents. You're on your own to protect your pages.
Granted, I have `disallow: /` in mine because I don't want my stuff scraped, but I still see Googlebot sometimes, and others, try to crawl my site.
It's not a very effective opt-out, because it requires the 'attacker' to honor the file's settings.
Feel free to enable it on your own server and watch the logs for a few days after sharing some links.
Granted, I have `disallow: /` in mine because I don't want my stuff scraped, but I still see Googlebot sometimes, and others, try to crawl my site.
It's not a very effective opt-out, because it requires the 'attacker' to honor the file's settings.
Feel free to enable it on your own server and watch the logs for a few days after sharing some links.