Comment by SalmonSnarker

Projects like this should respect that if a site's robots.txt contains a long list of Disallow entries for other AI scrapers that they are probably not welcome to scrape either.

(of course this project doesn't do that)