Discussion Ask HN: What do people use to prevent crawlers? What people use to prevent robots and crawlers from websites? 122 points jongi_ct 9 years ago 118 comments
