Comment by bananalychee

I self-host a few servers and have not seen significant traffic increases from crawlers, so I can't agree with that without seeing some evidence of this issue's scale and scope. As far as I know it mostly affects commercial content aggregators.