Earlier this month I wrote an ETL extractor using Capybara & headless browser (to work-around a lack of API - PS: only do that as a last resort!).
One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1].
PhantomJS won't work either (unless you use a custom build) [2].
I also tried with capybara-webkit, but no luck either [3].
The only driver which ultimately allowed me to download files at this point was Selenium + Firefox (with some tweaking on profile options browser.download.dir, browser.download.folderList = 2, browser.helperApps.neverAsk.saveToDisk set to the MIME type of files to be downloaded).
One thing I learned and wanted to warn about here is that Chrome headless currently doesn't support file downloads [1].
PhantomJS won't work either (unless you use a custom build) [2].
I also tried with capybara-webkit, but no luck either [3].
The only driver which ultimately allowed me to download files at this point was Selenium + Firefox (with some tweaking on profile options browser.download.dir, browser.download.folderList = 2, browser.helperApps.neverAsk.saveToDisk set to the MIME type of files to be downloaded).
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=696481
[2] https://github.com/ariya/phantomjs/issues/10052
[3] https://github.com/thoughtbot/capybara-webkit/issues/691