zopper
Joined 9 years ago
Karma
171
Submissions
19
Member since
Apr 18
Joined 9 years ago
More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything.
If you do this in two steps, you can always go back, change the scraper and quickly rerun it on historical data instead of re-crawling everything from scratch.