Comment by smetj

Offer the content, which is otherwise scraped, as a tarbal + diffs via a torrent or p2p network making it more efficient and cheaper for the crawlers to obtain the data.