Comment Comment by bediger4000 What is Ahrefs' business? Sure "SEO", but what does that mean, exactly?They seem to have a poorly coded web crawler. I wrote an "infinite web site" PHP program that Ahrefs reacted to very poorly: https://github.com/bediger4000/infinite-fake-websiteSince SEO is a major contributor to lousy search results, I'm now just trying to ban them all via robots.txt, but it doesn't seem mean enough. parent bediger4000 4 years ago
They seem to have a poorly coded web crawler. I wrote an "infinite web site" PHP program that Ahrefs reacted to very poorly: https://github.com/bediger4000/infinite-fake-website
Since SEO is a major contributor to lousy search results, I'm now just trying to ban them all via robots.txt, but it doesn't seem mean enough.