jackienotchan

How did you find your differentiation in a highly commoditized space? It's probably one of the most crowded spaces.

Even within YC, there are many competitors that do pretty much the same thing:

- Firecrawl

- BrowserUse

- Browserbase

- CloudCruise

- NotteLabs

- Intuned

- Expand.ai

- Reworkd

And then you have the extremely well-funded web retrieval players like Parallel and Exa.

How do you differentiate to all these?

Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).

Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?

Why is this not a Launch YC (or at least mention it?) since you seem to be part of the current batch?

The record/replay is definitely and interesting direction. The browser automation space is getting super crowded though (even within YC), so curious to hear how you differentiate from:

- BrowserUse

- Browserbase

- BrowserBook

- Skyvern

Congrats! Could this also be used to generate e2e test automations?

For scraping, how do you handle Cloudflare and Captchas? Do you respect robots.txt instructions of websites?

AI crawlers have lead to a big surge in scraping activity, and most of these bots don't respect any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits, user agents, etc.).

This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).

Does Webhound respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?

The first two quotes are from founders of:

- BrowserUse - Founded 2024

- Greptile - Founded 2023

The third quote is from a VC who has never founded a startup himself and has a clear interest in pushing founders to trade work-life balance for his own quick returns.

So none of these people worked on anything longer than 2 years. I wonder what will happen if we check back in 5–10 years. Will they still be doing and promoting 996, or will they be burned out and have changed their minds? Make your bets.

AI crawlers have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN (and experienced myself).

Do you have any built-in features that address these issues?

AI agents have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN.

Do you have any built-in features that address these issues?