> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file."
What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.
What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.