I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt.
If I actually wanted to restrict bots, it would be much easier to restrict googlebot because they actually follow the rules.
I don't disagree in principle that there should be an open index of the web, but for once I don't see Google as a bad actor here.
If I actually wanted to restrict bots, it would be much easier to restrict googlebot because they actually follow the rules.
I don't disagree in principle that there should be an open index of the web, but for once I don't see Google as a bad actor here.