Comment by tyropita

After a certain scale I think you can let clients do double-work and let the most common crawl data, among different clients, win.

And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.

There certainly are a lot of elegant ways to reduce spam for this particular problem imo.