so I have an upcoming side project that will heavily leverage scraping/screenshotting and I plan to use either headless chrome or slimerjs. It will be a Rails app, queueing up lots of sidekiq jobs, that I (presumably) will have calling a headless browser instance, probably running in a container. Ideally I'll run this either on Heroku or even better, just a DO droplet running dokku.
I found browserless.io and the supporting github project over the weekend, looks promising.
Who else is doing this? Is the architecture I just touched on the right way to go about doing this? Or should I have each sidekiq job spin up its own browser container for the scraping job?
Also, chrome vs firefox won't really matter in this instance, but reliability/performance/memory will...any recommendations?
I found browserless.io and the supporting github project over the weekend, looks promising.
Who else is doing this? Is the architecture I just touched on the right way to go about doing this? Or should I have each sidekiq job spin up its own browser container for the scraping job?
Also, chrome vs firefox won't really matter in this instance, but reliability/performance/memory will...any recommendations?