Comment by nuc1e0n

There is a * user agent that can be used to prevent all bots that honor robots.txt from crawling a site. Specific user agents can then be granted additional access afterward.