People with websites used to have a clear reason to allow bots to crawl and index our sites. Google and everyone sent us traffic. There was something of a trade off. Google has been slowly changing that trade by displaying more and more of our sites on google.com rather than sending people our way.
As far as I can see there's no sending people away from SearchGPT, it just gives answers. I can't see any reason to allow AI crawlers on my sites, all they do is crawl my site and slow things down. I'm glad that most of them seem to respect robots.txt.
As far as I can see there's no sending people away from SearchGPT, it just gives answers. I can't see any reason to allow AI crawlers on my sites, all they do is crawl my site and slow things down. I'm glad that most of them seem to respect robots.txt.