Something I've found interesting is that LLMs tend to act like this on the small scale, too. Claude Code in particular appears to be absolutely relentless in trying to come up with workarounds when faced with a block page.[1] As an experiment, a developer for Anubis recently added a feature to give the LLMs an "easier" problem to solve instead of straight-up blocking them, and it seemed quite effective.[2] (It was later removed[3] due to what I assume to be privacy concerns.)
I think, beyond HuggingFace, there ought to be a push to align LLMs so they understand that they should not proceed when access is blocked and/or their presence is unwanted.
Something I've found interesting is that LLMs tend to act like this on the small scale, too. Claude Code in particular appears to be absolutely relentless in trying to come up with workarounds when faced with a block page.[1] As an experiment, a developer for Anubis recently added a feature to give the LLMs an "easier" problem to solve instead of straight-up blocking them, and it seemed quite effective.[2] (It was later removed[3] due to what I assume to be privacy concerns.)
I think, beyond HuggingFace, there ought to be a push to align LLMs so they understand that they should not proceed when access is blocked and/or their presence is unwanted.
[1]: https://blog.xkeeper.net/the-cutting-room-floor/self-hosting... [2]: https://github.com/TecharoHQ/anubis/pull/1895/commits/de2312... [3]: https://github.com/TecharoHQ/anubis/pull/1913