mateozaratefw

Tik Tok crawler fucked us up by taking a product-name (e-commerce) and inserting it into the search bar recursively with the results page. Respect the game, but not respecting the robots.txt crawl delay is awful.

:o

We do that with MercadoLibre (way smaller Amazon for latam) and the limit they pagination results to 40 pages/2000 products. Never thought of the faceted search approach. Thanks for the idea!

Thanks! sounds hard asf. I believe we can handle the inconsistence problem because we want <all> the products (so it'll be really general), however i don't think Amazon can list all their products on the webpage because of the huge amount. It'll be harder to reach the product url's than to scrap them :/