Comment Comment by jsnell Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider.https://www.nerdcrawler.com/robots.txtThe domain serving the images is allowing everything:https://res.cloudinary.com/robots.txt parent jsnell 2 years ago
https://www.nerdcrawler.com/robots.txt
The domain serving the images is allowing everything:
https://res.cloudinary.com/robots.txt