Comment by jeremyjh

Understood. The comment I replied to suggests the crawlers should figure this out on their own.