Comment by ptasker

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily.

https://github.com/GoogleChrome/puppeteer