Comment by sunshowers

I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.