Scraping public web content is a really confusing situation.
My company, Datastreamer, has been in business for ten years indexing public web content (news + blogs). We focus primarily on "live" content. Content that publishes often.
The main challenge we've always had is that just because the content CAN be indexed doesn't necessarily mean you MAY index it.
A recent situation was around Craigslist vs 3taps:
https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.
Basically the issue doesn't evolve around WHO has copyright but who has copyaccess to the content.
So if you create an account on Acme.com... You still own the copyright to the content you post but Acme controls access. Not only that but the ToS that you sign gives them rights to your content including bulk sales.
This means that Acme can monetize the content that YOU create while actively preventing people from indexing it even that may be your intention.
This means that in 2018 a company like Google COULD NOT get started because websites would just not allow them to access your content.
I believe that when most people post public content on the Internet they intend it to be public including accessed by other search engines crawlers, etc.
Now we're in a horrible situation where just a few companies essentially own the Internet.
This is why Google can't index Facebook content or Twitter content even though it's public - they can't access it.
My company, Datastreamer, has been in business for ten years indexing public web content (news + blogs). We focus primarily on "live" content. Content that publishes often.
The main challenge we've always had is that just because the content CAN be indexed doesn't necessarily mean you MAY index it.
A recent situation was around Craigslist vs 3taps:
https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc.
Basically the issue doesn't evolve around WHO has copyright but who has copyaccess to the content.
So if you create an account on Acme.com... You still own the copyright to the content you post but Acme controls access. Not only that but the ToS that you sign gives them rights to your content including bulk sales.
This means that Acme can monetize the content that YOU create while actively preventing people from indexing it even that may be your intention.
This means that in 2018 a company like Google COULD NOT get started because websites would just not allow them to access your content.
I believe that when most people post public content on the Internet they intend it to be public including accessed by other search engines crawlers, etc.
Now we're in a horrible situation where just a few companies essentially own the Internet.
This is why Google can't index Facebook content or Twitter content even though it's public - they can't access it.