I have written crawlers for a while, most sites are unfriendly to anything other than Google (maybe they spare bing as well), some make it downright impossible without extensive work on evading their blocking techniques.
I expect (but can not confirm) this is the same reason why search engines like ddg and you.com also rely on bing's indexes.
Google's supremacy is here to stay. Only threat is bing or someone that comes with some excellent tech that bing incorporates (like openai) but actually works for search and is not just a LLM.
I expect (but can not confirm) this is the same reason why search engines like ddg and you.com also rely on bing's indexes.
Google's supremacy is here to stay. Only threat is bing or someone that comes with some excellent tech that bing incorporates (like openai) but actually works for search and is not just a LLM.