Comment by deceptionatd

Well-designed sitemaps and use of something like https://www.indexnow.org/ helps.

Cloudflare has Crawler Hints which works well IME: https://blog.cloudflare.com/crawler-hints-how-cloudflare-is-...