> What incentive would Google have to continue populating that index?
Presumably they would still want to run google.com and make money off of it.
> Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it?
No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to.
Updated based on edit of parent post:
> Maybe this is the problem that needs solving.
Why have websites waste the money to serve all those requests all over again? Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? I don't think website operators would be happy if there were a hundred more crawlers out there crawling as much as Google does now.
Presumably they would still want to run google.com and make money off of it.
> Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it?
No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to.
Updated based on edit of parent post:
> Maybe this is the problem that needs solving.
Why have websites waste the money to serve all those requests all over again? Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? I don't think website operators would be happy if there were a hundred more crawlers out there crawling as much as Google does now.