Comment by fooock

Yes, in the practice people sometimes don't want to be polite with webmasters, and choose not obey robots.txt rules. Thanks for the suggestion!