Comment by breadwinner

If it is publicly available information it is legal to scrape it, regardless of what robots.txt says.

See: https://www.webspidermount.com/is-web-scraping-legal-yes/