chadwebscraper

Just did - great chatting with you. Best of luck & looking forward to (potentially) hearing back!

First off, really sorry to hear that it didn't work.

Edit: it looks like you hit an edge case that I didn't see in testing. Happy to explain more, but it was skipping the extraction due to some pre-processing that craigslist was failing when it shouldn't have.

Would love if you want to try the tool again, but completely understand if not :)

This is actually a really interesting though - like an embedder with live data?

Shoot me your site and I can blacklist it

Thank you for the feedback - agreed.

It’s an extraction pattern for a certain site, so you can reuse it. Think a pattern to extract all forum posts - then using that on different pages with the same format. Like show new, show, new posts on HN.

Let me clarify, it just reindexes if the structured data changes, so it ignores layout changes. So it diffs the extractions.

Would be curious to try it out on your sites if you want to shoot me a few over - I can share my email.

It does use a browser to find dynamic content but does not afterwards.

So with APIs, it adjusts. For HTML layouts, it looks at the previous diffs to catch potential errors and then re-indexes.

lol it probably does unless their filtering is great

This is a good callout - I’ve tried my best thus far to limit the use of proxies unless absolutely necessary and then focus on reputable providers (even though these are a bit more pricey).

Definitely going to give this more thought though, thank you for the comment

I appreciate the response! I also agree - happy to add some clarity to this stuff.

Bot protection - this is handled in a few ways, the basic form bypasses most bot protections and that’s what you can use on the site today. For tougher sites, it solves the bot protections (think datadome, Akamai, incapsula).

The consistency part is ongoing, but it’s possible to check the diffs and content extractions and notice if something has changed and “reindex” the site.

100k URLs is a lot! It could support that, but the initial indexing would be heavy. It’s fairly resource efficient (no browsers). For scale, it’s doing about 40k/scrapes a day right now.

Appreciate the comments, happy to dive deeper into the implantation and I agree with everything you’ve said. Still iterating and trying to improve it.

Here’s how it works:

1. Paste a URL in, describe what you want

2. Define an interval to monitor

3. Get real time webhooks of any changes in JSON

Lots of customers are using this across different domains to get consistent, repeatable JSON out of sites and monitor changes.

Supports API + HTML extraction, never write a scraper again!