Comment by zeras

The index rate for each source automatically adapts based on the frequency of new content while maintaining a reasonable minimum frequency for sporadic sources.

The goal is to index content as quickly as possible without impacting each site's performance or causing automated defense systems to block the requests.