Comment by zmarty

A lot of news websites restrict any crawler other than Google. And this does not happen only via robots.txt.