Comment Comment by mcav This link corroborates that urllib2 respects robot.txt:http://stackoverflow.com/questions/3197299/urllib2-connectio... parent mcav 15 years ago
http://stackoverflow.com/questions/3197299/urllib2-connectio...