Comment by wibyweb

Thank you, my index is puny compared to yours and that is because I only index the page submitted by guests and don't go any further. The index is in the 10's of thousands. The hyperlink crawling feature was added and tested out specifically for the release, as I understand that some people will want to use that heavily and to build up a much larger index. My computers were super cheap and handle my puny index well, because its puny. I have no idea how well my approach compares to others in terms of performance.