Comment by ccgreg

> The CC CDX endpoint, index.commoncrawl.org, historically has been easily overwhelmed and unreliable

Use our Parquet index.

It's also worth noting that archive.org downloads all of our crawl data and adds it to the IA Wayback Machine.