Comment by stickfigure

> It's wrong to ignore a robots.txt.

It's not nice. It is, however, not illegal.

And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.