Comment by alphan0n

What about the CommonCrawl archives? That clearly show the same robots.txt that allows all, from September through December?

You’re a phony.