Discussion
How to crawl big websites with no sitemap?
The core of my startup is product data mining, and we have a really good suite of anti-bot tech. However, we need help crawling big sites like Amazon or eBay. Not because of the anti-bot stuff, but because they need a sitemap or trustworthy way of getting all the product URLs. Have any data-driven folks ever solved the problem of crawling big websites without a sitemap?