What are some examples of "most real world sites".
What are some examples of sites that are not "most real world sites".
Is HN a "real world site".
What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.)
Why not just delete or change the user-agent line in scrape.go before compiling.
(Personal experience: I have been successfully retrieving information from the www for decades without including a user-agent header. The number of sites I have found that require this header is relatively small. It does not rise to the level of "most".)
HN replies usually fail to include even a single example.
Similarly, do examples provided for scraper programs and libraries ever include "most real world sites".
What are some examples of sites that are not "most real world sites".
Is HN a "real world site".
What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.)
Why not just delete or change the user-agent line in scrape.go before compiling.
(Personal experience: I have been successfully retrieving information from the www for decades without including a user-agent header. The number of sites I have found that require this header is relatively small. It does not rise to the level of "most".)
HN replies usually fail to include even a single example.
Similarly, do examples provided for scraper programs and libraries ever include "most real world sites".