Is it fair to say that Crawler spoofing only works on sites that don't care?
Google recommends [1] checking the reverse DNS hostname of the source IP. I don't see any room for tricks there. So if you spoof a whitelisted Crawler simply by choosing the right values for some request headers, it means the publisher's web team is either clueless or doesn't care enough, no?
Google recommends [1] checking the reverse DNS hostname of the source IP. I don't see any room for tricks there. So if you spoof a whitelisted Crawler simply by choosing the right values for some request headers, it means the publisher's web team is either clueless or doesn't care enough, no?
[1] https://developers.google.com/crawling/docs/crawlers-fetcher...