hammadtariq
Joined 9 years ago
Joined 9 years ago
It lets site owners control automated access from AI crawlers and agents with per-site and per-post rules: allow, deny, teaser previews, and rate limits. It can also return 402 if you want to experiment with paid access.
If a crawler signs requests, the plugin can verify RFC 9421 HTTP Message Signatures. Most crawlers do not sign yet, but I wanted the path to be there.
It also publishes llms.txt, a JSON feed, and per-post Markdown endpoints.
Source and other integration surfaces (Node, Python, proxy): https://github.com/OpenBotAuth/openbotauth https://openbotauth.com/developers
If you run a content site or a crawler team, I would love feedback on signing defaults and replay protection expectations.
This week we shipped the following packages for the website operators that verify signed HTTP requests so crawler and agent traffic can be tied to a private/public keypair. Once you can verify identity, you can do allowlists, rate limits, analytics, and policy enforcement at the origin or edge.
- Node verifier middleware: @openbotauth/verifier-client (Express and Next.js style) - Python verifier: openbotauth-verifier (FastAPI and Flask) - Zero code reverse proxy: @openbotauth/proxy - Registry signer utilities: @openbotauth/registry-signer (Ed25519 + JWKS) - Test crawler and key generation: @openbotauth/bot-cli - WordPress plugin (pending WP.org review if you run WP)
Links: - NPM org: https://www.npmjs.com/org/openbotauth - PyPI: https://pypi.org/project/openbotauth-verifier/
We would love feedback on the threat model, replay protection approach, or how to build sub-agent identities using X509 certs.