On a vaguely related note, I hacked together a WayBack machine type thing on top of Common Crawl in like 50 lines of Python. So it transparently lets you load any page directly from the dumps (it loads only those few bytes from the crawl dumps, and gunzips them).
It was really easy, and I was surprised it apparently hadn't been done before.
The only problem is the index IP blocks you if you query it too often, and it gets queried once for every resource on the page.
I've been informed that there are ways around this (download an index?), does anyone know more about that?
It was really easy, and I was surprised it apparently hadn't been done before.
The only problem is the index IP blocks you if you query it too often, and it gets queried once for every resource on the page.
I've been informed that there are ways around this (download an index?), does anyone know more about that?