Comment by ihartley

You can use something like scrapy-playwright[0] to run a headless browser framework as your download handler. I think there are versions for some of the other headless systems, if you prefer those.

[0] https://github.com/scrapy-plugins/scrapy-playwright