Comment

Comment by crowcroft

Another common unintended consequence I've seen is conflating crawling and indexing with regards to robots.txt.

If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index.

If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled!

Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.

parent crowcroft 1 year ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton