T twitter.com Robots.txt for the NYT has a specific exclusion for an 1996 news article 270 points mikeortman 5 years ago 100 comments
