Set your site to 401 unauthorized with a basic challenge if an auth header isn’t sent, and set the auth description to “Enter anything to proceed. All human access is authorized. Unauthorized non-human access is prohibited.”. Crawlers can’t parse the instructions and will deadstop on them, while people will shrug and enter any password, which will work.
Anubis is also viable and popular, but it lacks the legal threat to AI of being able to file a federal hacking claim against a scraper’s unauthorized intrusion if they code their scraper to transmit an empty/invalid/valid authentication header.
Anubis is also viable and popular, but it lacks the legal threat to AI of being able to file a federal hacking claim against a scraper’s unauthorized intrusion if they code their scraper to transmit an empty/invalid/valid authentication header.