Comment Comment by szczepano there is simple solution for gentlemens:robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this websiteright now we need to do this:User-agent: *Disallow: / parent szczepano 1 year ago
robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website
right now we need to do this:
User-agent: *
Disallow: /