Comment by bobpaw

TIL about the Crawl-delay directive. Although it seems that most honest bots move slower and dishonest bots will learn to.