Comment Comment by gigaArpit Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare. parent gigaArpit 1 year ago