Discussion Ask HN: Best practices for ethical web scraping? Hello HN!As part of my learning in data science, I need/want to gather data. One relatively easy way to do that is web scraping.However I'd like to do that in a respectful way. Here are three things I can think of:1. Identify my bot with a user agent/info URL, and provide a way to contact me 2. Don't DoS websites with tons of request. 3. Respect the robots.txtWhat else would be considered good practice when it comes to web scraping? 270 points aspyct 6 years ago 107 comments
As part of my learning in data science, I need/want to gather data. One relatively easy way to do that is web scraping.
However I'd like to do that in a respectful way. Here are three things I can think of:
1. Identify my bot with a user agent/info URL, and provide a way to contact me 2. Don't DoS websites with tons of request. 3. Respect the robots.txt
What else would be considered good practice when it comes to web scraping?