Web Bot Auth: How AI Agents Prove Their Identity to Websites (RFC 9421 and Visa TAP)
Key takeaways
- Agents often get blocked because they are anonymous, not because they are unwelcome: bot defenses cannot tell a legitimate agent that buys for a real customer from abusive automation.
- Web Bot Auth gives an agent a cryptographic identity: the agent signs each HTTP request with HTTP Message Signatures (RFC 9421) and publishes its public key in a key directory that verifiers can fetch.
- Verification is a signature check against a published key, so a merchant or CDN gets a cryptographic answer to "which agent is this" instead of a guess from IP addresses and user agent strings.
- Web Bot Auth proves which operator sent a request. It does not prove that the operator is trustworthy, that a human approved a purchase, or that a payment credential is valid.
- Visa's Trusted Agent Protocol works at a different layer: it gives merchants a way to recognize agents in a payments context, and it builds on the same signature approach.
Web Bot Auth lets an AI agent prove which agent it is on every HTTP request. The agent signs each request with a private key using HTTP Message Signatures (RFC 9421) and publishes the matching public key in a key directory, so merchants and CDNs can verify the caller cryptographically instead of guessing from IP addresses. Cloudflare has driven much of the work, and the specifications are developed in the open at the IETF.
This page describes public standards and drafts at a general level. Drafts change, so check the current IETF documents and your CDN provider's documentation before you implement.
Why do AI agents get blocked at merchants?
AI agents get blocked at merchants because bot defenses cannot tell a legitimate agent from abusive automation, so blocking unidentified automation is the safe default. Most merchants do not reject agents on principle. They deploy bot defenses because they have real abuse problems.
- Anonymity is the default: An HTTP request from an agent carries nothing trustworthy about who sent it. A user agent string is a self-report that anyone can copy, so it cannot be the basis of a trust decision.
- Bot defenses are built for abuse: Providers such as Cloudflare, Akamai, and DataDome exist to stop credential stuffing, scraping, inventory hoarding, and card testing. Against unidentifiable automation, a block is the correct default.
- The available signals are indirect: IP reputation, request rates, and behavioral heuristics infer intent from side effects, not from identity. They produce false positives, and legitimate agents land in them.
- A merchant cannot allow what it cannot name: A merchant that wants agent-driven orders has no mechanism to allow specific agents when no agent can prove its identity.
The blocking problem is an identity problem. You cannot allowlist an anonymous caller.
What is Web Bot Auth?
Web Bot Auth is a set of conventions that gives automated clients a verifiable cryptographic identity at the HTTP layer. It builds on the published RFC 9421 standard and on IETF drafts, and Cloudflare has driven much of its development. It has two parts.
- Signed requests: The agent signs each outbound HTTP request with a private key using HTTP Message Signatures, defined in RFC 9421, and attaches the signature and its metadata as HTTP headers.
- A published key directory: The agent operator publishes its public keys at a well-known location, as the HTTP message signatures directory draft describes, so any verifier can fetch the key that a signature references and check it.
The design is simple on purpose. There is no new transport, no handshake to negotiate, and nothing for a merchant to install. A verifier that can fetch a URL and check an Ed25519 signature can verify an agent.
How do HTTP Message Signatures (RFC 9421) work?
HTTP Message Signatures (RFC 9421) work by signing a canonical string built from selected parts of an HTTP request, so a recipient can rebuild that string and verify the signature with the signer's public key. The steps are below.
- 1
Choose the covered components: The signer selects which parts of the request the signature covers. For agent identity these include derived components such as the authority or the target URI, so the signature is bound to the specific request.
- 2
Build the signature base: The covered components are serialized into a canonical string, in a defined order, together with signature parameters such as the key identifier, the algorithm, a creation timestamp, an expiry, and a tag that identifies the purpose.
- 3
Sign it: The signer produces a signature over that canonical string with its private key. Ed25519 is a common choice for Web Bot Auth because the keys and signatures are small and verification is fast.
- 4
Attach two headers: The request carries a Signature-Input header that describes what was signed and with which key, and a Signature header that carries the signature bytes.
- 5
Reconstruct and check: The recipient rebuilds the same canonical string from the request it received, fetches the public key that the key identifier names, and verifies the signature. If the request was altered or the key does not match, verification fails.
The signature is bound to request components and carries a creation time and an expiry, so a captured signature cannot be usefully replayed against a different request or reused without limit.
What is a key directory, and why does publishing keys matter?
A key directory is a list of an agent operator's public keys, in JSON Web Key form, served at a well-known HTTPS location so that any verifier can fetch and cache them. A signature has meaning only if the verifier can get the correct public key, and the directory supplies it.
- It makes verification permissionless: Any merchant or CDN can verify a signed request without a prior relationship with the agent operator, a shared secret, or an onboarding process.
- It ties identity to a domain: The directory is served over HTTPS from the operator's own domain, so the key is bound to a named, accountable entity and not to an anonymous client.
- It supports key rotation: Each signature references a specific key identifier, so an operator can publish a new key and retire an old one without breaking verifiers.
- It scales without coordination: One published directory serves every verifier on the internet. This makes the model workable across a very large number of websites.
How does a merchant or CDN verify a signed agent?
A merchant or CDN verifies a signed agent by fetching the public key that the signature names, checking the signature against the request as received, and then applying its own policy to the verified identity.
- 1
Notice the signature headers: The incoming request carries Signature-Input and Signature headers. They tell the verifier that this client claims a verifiable identity.
- 2
Resolve the key: The verifier reads the key identifier from the signature parameters and fetches the matching public key from the operator's published directory, typically with a cache.
- 3
Verify the signature: The verifier rebuilds the signature base from the request as received and checks the signature against the public key, along with the creation and expiry times.
- 4
Apply policy: Verification produces an identity, not a decision. The verifier then applies its own policy: allow, rate limit differently, require more steps, or deny. This often happens at the CDN before the request reaches the merchant's application.
What does Web Bot Auth prove, and what does it not?
Web Bot Auth proves which agent operator signed a request and that the signed parts were not changed in transit, and it does not prove trustworthiness, human consent, or payment validity. Other layers answer those questions.
| Question | Does Web Bot Auth answer it? |
|---|---|
| Which agent operator sent this request? | Yes, cryptographically, through a signature over a published key |
| Was this request changed in transit? | Yes, for the components the signature covers |
| Is this a replay of an earlier signature? | Largely mitigated, through covered components and expiry |
| Is this operator trustworthy? | No, that is the verifier's policy decision |
| Did a human authorize this purchase? | No, that is the consent layer, such as an approved purchase intent |
| Is the payment credential valid? | No, that is the card network layer, such as network tokens and issuer authorization |
An agent purchase needs all of these layers. Verifiable identity gets the agent served, an approved intent shows that a human authorized the purchase, and a valid credential gets the payment authorized.
How does Web Bot Auth relate to Visa's Trusted Agent Protocol?
Web Bot Auth and Visa's Trusted Agent Protocol are complementary: Web Bot Auth proves which operator signed a request, and the Trusted Agent Protocol helps merchants recognize trusted agents in a payments context. Visa developed the Trusted Agent Protocol with Cloudflare, and it builds on the same HTTP Message Signatures approach.
A merchant can use both signals. The signature tells the merchant that the caller is who it claims to be. Agent recognition in a payments context tells the merchant that the caller is a known agent that acts for a consumer, not only a well-behaved crawler. Card networks also offer agent payment programs, such as Visa Intelligent Commerce and Mastercard Agent Pay, that address the credential layer.
What does an agent operator need to implement Web Bot Auth?
An agent operator needs a signing key pair, a signing step on every outbound request, a published key directory, a rotation process, and tests against the CDNs that enforce bot policy.
- Key generation and custody: Generate an Ed25519 key pair and keep the private key in a secure key store, never in application code or logs.
- A signing path: Sign every outbound request that the agent makes, with the covered components, creation time, expiry, and tag that the verifier expects.
- A published directory: Serve the public keys over HTTPS at the well-known location on a domain that the operator controls.
- Rotation: Publish a new key before it is used, sign with it, and remove the old key only after its signatures have expired.
- Verification testing: Confirm with each CDN provider that your signatures verify, and register with provider programs where they exist.
Where is Web Bot Auth heading?
Web Bot Auth is moving toward a normal, checkable property of a request: RFC 9421 is a published standard, the directory conventions are moving through IETF drafts, and CDN providers have started to verify signatures at the edge.
For builders, the consequence is direct. As verification becomes standard, the split will be between agents with a verifiable identity and agents without one. Websites can make deliberate policy decisions only for the first group.
Frequently asked questions
What is Web Bot Auth?
Web Bot Auth is an emerging set of conventions that gives automated clients a verifiable cryptographic identity at the HTTP layer. The agent signs each request with a private key using HTTP Message Signatures (RFC 9421) and publishes the matching public key in a key directory, so merchants and CDNs can verify which agent is calling instead of guessing from IP addresses and user agent strings.
What is RFC 9421 and why does it matter for AI agents?
RFC 9421 is the published IETF standard for HTTP Message Signatures. It defines how a signer selects which parts of a request to cover, builds a canonical signature base with a key identifier and expiry, and attaches Signature-Input and Signature headers. For AI agents it is the mechanism that turns an anonymous request into a verifiable, attributable one.
Why do AI agents get blocked at checkout?
AI agents get blocked at checkout mostly because they are anonymous, not because merchants reject agents on principle. Bot defenses from providers such as Cloudflare, Akamai, and DataDome stop scraping, credential stuffing, and card testing, and unidentifiable automation looks like that traffic. Verifiable identity through Web Bot Auth makes a selective merchant policy possible.
What is a key directory in Web Bot Auth?
A key directory in Web Bot Auth is a set of an agent operator's public keys, in JSON Web Key form, served over HTTPS at a well-known location on the operator's domain. A verifier fetches the key that a signature names and checks the signature against it, without a prior relationship with the operator.
How is Visa's Trusted Agent Protocol related to Web Bot Auth?
Visa's Trusted Agent Protocol, developed with Cloudflare, builds on the same HTTP Message Signatures approach as Web Bot Auth. Web Bot Auth proves which operator signed a request. The Trusted Agent Protocol helps merchants recognize trusted agents that act for consumers in a payments context. The two are complementary.
Does Web Bot Auth prove that a user authorized a purchase?
No. Web Bot Auth proves which agent operator sent a request and that the signed parts of the request were not changed. Proof that a human approved a specific purchase belongs to the consent layer, typically an approved, line-item purchase intent confirmed with a passkey or biometric before anything is bought.
Related articles