Comment by yellowapple

I'm surprised the Wayback Machine doesn't account for that by figuring "well our archive of this predates the current robots.txt, so let's not remove it".