I am trying to create a tool for myself that can crawl a few websites that I usually go on to compare the price of the same item. They have no APIs.
A couple questions:
1. is this LEGAL?
2. if I am crawling, what is the best way to approach this? does each website's crawling mechanism have to be manually written since they are unique or is there some strategy for scale if i need to expand the number of sites I crawl through in the future?
I am trying to create a tool for myself that can crawl a few websites that I usually go on to compare the price of the same item. They have no APIs.
A couple questions:
1. is this LEGAL? 2. if I am crawling, what is the best way to approach this? does each website's crawling mechanism have to be manually written since they are unique or is there some strategy for scale if i need to expand the number of sites I crawl through in the future?
Thank you!
-F75