Comment Comment by johnny99 Compliance with robots.txt is and always has been voluntary, and many crawlers have long ignored it, including Archive.org. parent johnny99 7 years ago
