Comment

Comment by chr13

>"buying 1Gbps uplink is trivial"

1 Gbps dedicated uplink is not trivial everywhere in the world. You also need more quota. Facebook easily crawled more than 1 TB of data during an hour. The more the better may not be best solution here. The higher the uplink, faster Facebook may crawl. So instead of transfering 1 TB in an hour for a 1 Gbps uplink, the transfer will be 10 TB if you have 10 Gbps. This will also depend how much bandwidth they allow their crawler, there must be an upper limit though.

parent chr13 12 years ago

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton