Comment by nequo

You could probably build a mirror with `wget -r` without writing a crawler.