Comment Comment by botsbreeder AhrefsBot respects robots.txt. Details are here: https://ahrefs.com/robot parent botsbreeder 4 years ago