Comment by josefresco

Have you ever been responsible for the performance and security of a publicly accessible web server? I'll accept robots indexing my content if they play nicely. Unfortunately most do not, even from major vendors.