Comment

Comment by DougWebb

Possible, but seems unlikely. To set that up the WSJ website would have to allow Googlebot access while denying others. Any filtering based on the url or HTTP headers would be discovered and abused by others. An approach based on a security token or IP filter could work, but would be un-managable on Google's side because of the scale of their spidering operation. It would be much more effective for them to use their position to force the WSJ to be an open website, or to accept that their paywalled content does not get indexed.
parent DougWebb 9 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton