Comment by rgrieselhuber

It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.