> Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...)
Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.
> Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...)
Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.