Comment by xmcp123

I have been writing bots and scrapers for almost 20 years.

Not once have I had someone I work for in this capacity even ask about robots.txt