Comment Comment by walski IA has not honored robots.txt for the better part of a decade now.https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... parent walski 1 year ago
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...