When doing this in the past, I settled on an sqlite database with one table that stores the compressed html (gzip or lzma) along with other columns (id/date/url/domain/status/etc.)
Also made it easy to alert on when something broke (query the table for count(*) where status=error) and rerun the parser for failures.
Also made it easy to alert on when something broke (query the table for count(*) where status=error) and rerun the parser for failures.