Comment by trinari

robots.txt is a way of leaving the door unlocked but kindly asking bots to stay outside.