Verify Rapidflare's crawler

Rapidflare's crawler does two kinds of fetching: ingestion of the websites, sitemaps and URL lists customers connect as knowledge sources, and research crawling of publicly available product documentation for Rapidflare's product catalog. You can confirm that crawl traffic genuinely comes from Rapidflare in two independent ways: cryptographic request signatures (Web Bot Auth) and our published egress IP addresses.

Web Bot Auth request signatures

Every request Rapidflare ingestion sends to a configured knowledge source is signed with HTTP Message Signatures (RFC 9421) following the Web Bot Auth architecture. Each request carries three extra headers:

Signature-Agent: "https://docs.rapidflare.ai"
Signature-Input: sig1=("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";nonce="…";tag="web-bot-auth"
Signature: sig1=:…:

The signature is produced with a private key only Rapidflare holds, covers the host being crawled, and expires within minutes — so it cannot be replayed against another site or reused later. The matching public keys are published in our key directory:

The keyid in each request's Signature-Input is the RFC 7638 thumbprint of the JSON Web Key in that directory, which is how a verifier picks the right key.

What this means for your bot protection

If your site sits behind a platform that verifies Web Bot Auth — Vercel Bot Protection and Cloudflare Bot Management both do — Rapidflare is recognised as a verified bot and passes their challenges without any configuration on your side. There is nothing to install and no shared secret to exchange: verification happens entirely from the signature and our public key directory.

If you run your own verification, validate the signature as described in the Web Bot Auth draft: fetch the key directory, select the key by keyid, rebuild the signature base from @authority and signature-agent, and check the Ed25519 signature and the created/expires window.

Why not just check the User-Agent?

A User-Agent header is a claim anyone can make, and the crawler may present different User-Agent strings depending on what the target site accepts. Treat the signature (strongest) or the source IP as the identity check, not the User-Agent.

Source IP addresses

All Rapidflare crawl traffic — customer-configured ingestion and catalog research alike — originates from four static IP addresses:

34.28.22.66
34.30.154.174
34.41.228.67
35.255.80.253

A request claiming to be Rapidflare from any other address is not ours. If your content requires an IP allowlist to be reachable at all, see IP Allowlisting for how to configure it and verify it works.

What the crawler reads

Most Rapidflare crawl traffic is customer-directed ingestion: fetching the specific URLs, sitemaps and sites a customer has connected in their admin dashboard, on the sync schedule they set for each source. Ingestion does not follow links beyond what a configured source defines, and this is the traffic that carries Web Bot Auth signatures.

Separately, Rapidflare performs catalog research crawling: reading publicly available product documentation — manufacturers' datasheets, errata and product pages — to build and maintain the structured product-catalog data behind Rapidflare's recommendation and cross-reference agents. This traffic reads public pages only, identifies itself with a RapidflareCatalogBot User-Agent that links back to this site, and originates from the same egress IP addresses listed above.

Crawled content powers retrieval and catalog features for Rapidflare's customers.

If you have questions about crawl behaviour on your site, or would like your site excluded from catalog research, contact your Rapidflare representative or reach us through rapidflare.ai.