Comment by ashkankiani

Use a proper rate limiting scheme, obviously. You're using access patterns associated with human volumes, so you can only ever scrape at human speeds (except that you get the advantage of constantly saturating the pipe and running 24/7)