Comment

Comment by szczepano

there is simple solution for gentlemens:

robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website

right now we need to do this:

User-agent: *

Disallow: /

parent szczepano 1 year ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton