Good performance matters if you have decent networking infrastructure or your server has limited resources.
Bandwidth and IP limits are the most common bottle necks, but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies [1].
Bandwidth and IP limits are the most common bottle necks, but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies [1].
[1] http://go-colly.org/docs/best_practices/distributed/