Ultimately I think it’s because the internet is just too big! Google were only able to do it because they had the right algorithm early in the age of the internet. They were then able to grow with it to achieve the scale required. Starting from scratch now on a general internet search engine would be close to impossible without 10s, if not 100s, billions of investment. And you would need that to build the index before even beginning to be competitive. No one is making bets that big on search, especially when the online advertising industry (which is the only way to fund it currently) is in danger of massive regulation.
I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible.
You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine.
Edit:
Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search.
We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.
I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible.
You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine.
Edit:
Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search.
We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.