Comment

Comment by bhartzer

Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive.

Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are removed from the index, block the google IPs.

parent bhartzer 7 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton