Comment by simonw

A trick I think would be useful to include here is running scrapers in GitHub Actions that write their results back to the repository.

This is free(!) to host, and the commit log gives an enormous amount of detail about how the scraped resource changed over time.

I wrote more about this trick here: https://simonwillison.net/2020/Oct/9/git-scraping/

Here are 267 repos that are using it: https://github.com/topics/git-scraping?o=desc&s=updated