Comment Comment by Jskewel The big USA ones do, and it would be madness for them to do otherwise.But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks. parent Jskewel 26 days ago
But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks.