You could store the data in a git repo per domain, so that implicit de-duplication happens on re-visits & for shared resources.
You could have a raw dir (the files you receive from the server) and a render dir that consists of snapshots of the DOM + CSS with no JS & external resource complexity.
When the global archive becomes too big, history could be discarded from all the git repos by discarding the oldest commit in each repo, and so on.
SOLR is probably the right tool for the index but there is something undeniably appealing about staying in the pure file paradigm - you could use sqlite's FTS5 module to do that too.
You could have a raw dir (the files you receive from the server) and a render dir that consists of snapshots of the DOM + CSS with no JS & external resource complexity.
When the global archive becomes too big, history could be discarded from all the git repos by discarding the oldest commit in each repo, and so on.
SOLR is probably the right tool for the index but there is something undeniably appealing about staying in the pure file paradigm - you could use sqlite's FTS5 module to do that too.