Comment by Chief_Searcha

Thank you! Both, I use common crawl as the bulk, but it has long delays so I need to crawl myself too. I'm planning on upgrading my own crawling capabilities but not yet ready since it will require an investment. I also have plans to have a smarter crawl system which I may implement in the next half year.