Comment by ejr

These are often used on crawlers, bots - not necessarily of the harmful variety - and other automated retrieval, transfer and storage scripts. They're very useful, but highly specialised for those particular tasks and often not needed on traditional web applications. Of course, these are often abused for retrieving malware payloads as well so enable at your own risk.