Why? Why is a random response code not good enough?
Rather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out.
Just a few days ago, an OSS project that I participate in had its wiki scraped, that looked like some random mac user agent, but suddenly from around the world, everyone looking at the wiki history, reading with crazy speed...
Rather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out.
Just a few days ago, an OSS project that I participate in had its wiki scraped, that looked like some random mac user agent, but suddenly from around the world, everyone looking at the wiki history, reading with crazy speed...