Comment

Comment by cube00

> Till helps you circumvent detected as a web scraper by identifying your scraper as a real web browser. It does this by generating random user-agent headers and randomizing proxy IPs (that you supply) on every HTTP request.

How did we start with consent using robots.txt and end up here?

A good neighbor doesn't use circumvention.

parent cube00 5 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton