This seems unnecessarily complicated and antagonistic. Mostly just to publicly shame a bunch of people.
At Reddit we originally blocked a couple of crawlers but then realized how pointless that was. The entire robots file[0] is basically now just for google. All of the restrictions are enforced on the server side because there were so many bad bots it didn’t matter if we listed them or not.
At Reddit we originally blocked a couple of crawlers but then realized how pointless that was. The entire robots file[0] is basically now just for google. All of the restrictions are enforced on the server side because there were so many bad bots it didn’t matter if we listed them or not.
[0] https://www.reddit.com/robots.txt