dmn001

I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.

By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd party library then data deletion does not matter too much either if you configure it properly and backup the data - html/json data compresses really well using lzma2 in 7-zip.

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging.

A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without downloading the entire site over again.

On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.

Should be fixed now.

No. Most websites don't do this.

It's extremely rare to be ip-blocked by any website just for using the Google's user agent from a non-specific range. IP's get re-used and you can switch to a new one easily, so it's really not common or good practice for this to happen.

SEEKING WORK - England,UK/Remote

Over 7 years experience with Python software development, cloud services, data mining, web crawling, databases. Want to extract or crawl data from a website such as business listings, sports data, government data, site directories, either one-off, periodically, or in real-time? Contact me via email: dmn001(at.[gmail

The first part seems like a very long-winded way to say "don't use the default user agent".

The captcha was unusually simple to solve, in most cases the best strategy is to avoid seeing it in the first place.

Right, if you're just matching a substring in a list of strings then you don't need regex for that.

Also, lxml is over 20x faster than BeautifulSoup, although it doesn't really matter with such a low frequency of requests, it's something to note if you had another project that required parsing e.g. 1,000's of html pages.

SEEKING WORK - UK/Remote

Over 7 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.