Comment by IvanAchlaqullah

At this rate, robots.txt will be ignored by bots if it's not already.

Just like ads company completely ignoring Do Not Track because it's always set to deny / opt-out by the browser. It's already been fixed, but the damages is already done.

https://en.wikipedia.org/wiki/Do_Not_Track#Internet_Explorer...