Discussion

How to crawl big websites with no sitemap?

The core of my startup is product data mining, and we have a really good suite of anti-bot tech. However, we need help crawling big sites like Amazon or eBay. Not because of the anti-bot stuff, but because they need a sitemap or trustworthy way of getting all the product URLs. Have any data-driven folks ever solved the problem of crawling big websites without a sitemap?
9 points mateozaratefw 2 years ago 9 comments

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton