safehuss
Joined 4 years ago
Karma
30
Submissions
13
Member since
Sep 14
Joined 4 years ago
The act of slowing down to ensure every line makes sense to me is one of the most valuable things for my learning.
For the more challenging scrapes, would highly recommend using the Chrome Devtools MCP to be able to attach the network requests, being made by the browser to the site, as context for your agent/LLM chat - this approach really helped me to write a solid API-based scraper (also using curl_cffi) and bypassed the old tedious playwright-based approach I used to rely on.
That's assuming the crawlers haven't ingested it all already...