Ian_Kerins

A lot of the discussion around the /crawl endpoint seems to miss a key detail in the docs. The crawler explicitly identifies itself as a bot, respects robots.txt, and does not bypass CAPTCHAs, WAF rules, or Cloudflare Bot Management.

So technically it’s a nice managed crawling system, but in practice it only works on sites that already allow bots to crawl them. For many real-world data extraction use cases, the problem isn’t crawling infrastructure, it’s dealing with sites that actively block bots. In those cases you still need traditional scraping approaches.

Interesting take on it. Some people probably wouldn't like to be called soft but there is likely some truth to it.

I feel it really comes down to priorities.

Scraping has always been a means to a end for most companies. Get data and then use it for something valuable. Before getting the data was easy, but now it is getting increasingly harder.

I think the key here is highlighting the fact that the time of cheap/easy/low skilled access to web data is ending. Companies either need to skill up on understanding how to bypass anti-bots or pay someone else to do it for them and they focus on the data.

One of the main ideas, we explored here is how scraping has shifted from being mainly a technical challenge to an economic one:

- Infrastructure and proxies have gotten cheaper, but anti-bot defenses have evolved fast.

- Because of that, the real cost of scraping is now the cost per successful result, and spikes of 5x–20x can happen when defenses tighten.

- The bottleneck today isn’t just “can you scrape it?”, it’s whether you can do it profitably and efficiently.

I’d love to hear how folks here are dealing with rising scraping costs or what strategies have worked when data value doesn’t obviously outweigh defense costs.

We just dropped the State of Web Scraping 2025 report. TL;DR: scraping is scaling—fast.

- Market boom: Web scraping is growing 15% YoY and projected to hit $13B by 2033. Web data is now a real asset class. The gold rush is on.

- AI + scraping: LLMs are getting surprisingly good at generating spiders, debugging selectors, and auto-healing. Still brittle, but improving fast. 2025 might be the year of the “self-healing” scraper.

- Bot wars intensify: Anti-bot tech is getting aggressive, Cloudflare, decoy pages, forced JS rendering, login walls. Scraping popular sites is now high-effort and high-cost.

- Proxy market shake-up: Residential/mobile proxy prices have dropped 25–50% in the last year, thanks to scrappy newcomers. But domain-level pricing is rising, creating more complexity and less transparency.

- Legal landscape: Lines are getting clearer: public data is generally safe; behind logins is risky. AI crawlers are under increasing scrutiny, and enforcement is likely to tighten.

- Scraping stack evolution: New tools are focusing on stealth, AI assistance, and integration into real data pipelines. The modern scraping stack looks more like infrastructure than hacked-together scripts.

Big picture: 2025 is shaping up to be a turning point. Smarter scrapers, tougher competition, higher stakes.

It is this type of attitude that is why websites are becoming so aggressive in blocking web scrapers.

Being an "ethical web scraper" is about your own ethics, not abusing other peoples servers/data, and preserving the open internet for everyone.

Yes, if you slam someones website you will probably get the data you want but it will damage it for the collective. If everyone just slammed websites then:

1) the website owners will just get pissed off and either shut data behind logins or make it inaccessible. This is basically what happened with LinkedIn.

2) make it much harder and costlier to scrape by using more advanced anti-bots, requiring everyone to use more expensive residential proxies and headless browsers to get past them.

Web scraping can be a burden for websites, so everyone should approach it in as responsible and ethical way as possible.

Thanks for sharing it. It currently works with Scrapy & Python Request scrapers, will be launching SDKs for Node, Puppeteer, etc. soon.