Is there a good way to combine Scrapy framework (retry, rate limiting,..) with a headless browser such as selenium (to get full js-loaded client-side data)?
When I had to do it I ended up duplicating each page request twice. Once for scrapy and once again with selenium.
When I had to do it I ended up duplicating each page request twice. Once for scrapy and once again with selenium.