Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it.
How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl?
Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl?
Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.