Comment by DaSHacka

Depending on the service, you may not want it to be indexed anyway (IE login pages) or, if you do, most commercial crawlers have methods to allow site owners to verify that the traffic is a legitimate crawler and not an impersonator, see:

https://developers.google.com/crawling/docs/crawlers-fetcher...

(Though restricting the Google UA to only legitimate Google IPs seemingly would break Brave's crawler, though that's their own fault: https://safe.search.brave.com/help/brave-search-crawler/)

https://duckduckgo.com/duckduckgo-help-pages/results/duckduc...

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...

https://kagi.com/bot