Comment by senko

A crawler that respects robots.txt is useless in practice since many sites only allow Googlebot and maaaybe Bing - by name.