Comment by gtirloni

The training data could include internal docs that describe how it ignores or not the robots.txt file.