Comment

Comment by tangue

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper.

Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate.

You could say that scraping you're website is FORBIDDEN, but come on : if Airbnb can rent houses, I can scrap you site.

parent tangue 10 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton