Comment Comment by threeseed I wasn't aware. Can you please update Wikipedia then: https://en.wikipedia.org/wiki/Robots.txtMaybe also get Google to update their docs: https://developers.google.com/search/docs/crawling-indexing/... parent threeseed 1 year ago
Maybe also get Google to update their docs: https://developers.google.com/search/docs/crawling-indexing/...