Comment by eru

They explicitly allow eg the Google crawler. Not sure about others.