You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it:
* For every Git commit, fetch the version of every file in the repository at that commit.
* See git blame for every file at every commit.
* Attempt to download the archive of each repo at every commit.
* Run every possible pull request search filter combination.
* Run every possible issue search filter combination.
* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.
Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.
* For every Git commit, fetch the version of every file in the repository at that commit.
* See git blame for every file at every commit.
* Attempt to download the archive of each repo at every commit.
* Run every possible pull request search filter combination.
* Run every possible issue search filter combination.
* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.
Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.