What is AutoRAG?
Direct Answer: AutoRAG is Cloudflare's AI search solution for indexing and querying data.
AutoRAG, now known as AI Search, is Cloudflare's managed search service that automatically creates a continuously updating index of connected data sources. It supports natural language queries and integrates with Cloudflare's developer platform tools. The service enables retrieval-augmented generation (RAG) patterns for building enterprise search, natural language search, and AI-powered chat.
User-Agent Identification
The following user-agent strings identify AutoRAG in your live traffic data:
Cloudflare-AutoRAG (https://developers.cloudflare.com/autorag; autorag@cloudflare.com)
robots.txt Rules for AutoRAG
Respects robots.txt: Yes
Use the following robots.txt rules to control AutoRAG access:
# Block AutoRAG
User-agent: Cloudflare-AutoRAG
Disallow: /
# Allow AutoRAG
User-agent: Cloudflare-AutoRAG
Allow: /Robots.txt is a directive, not a barrier
Cloudflare states that AutoRAG respects robots.txt. However, configuration mistakes, caching delays, and edge cases mean your directives may not always be followed as expected. Live traffic verification confirms whether AutoRAG actually obeys your rules in practice.
Need continuous verification across 500+ bots? Can AI See It automates this.
Crawl Behavior
Frequency:Continuous
Request Pattern:Automatically And Continuously Index Data Sources
Official Documentation Quotes
"AI Search (formerly AutoRAG) is Cloudflareâs managed search service."
Why track AutoRAG traffic?
Measure what Cloudflare gives back. AutoRAG fetches your content to power AI search — but does Cloudflare send traffic in return? Track referral visits from Cloudflare's AI search products relative to how much content AutoRAG consumes.
See which pages fuel AI-generated answers. AutoRAG fetches your content in real time to build search results for Cloudflare's AI search product. Tracking these requests shows you which URLs are being used as sources.
Measure referral traffic from AI search. When AutoRAG cites your page in an AI-generated answer and a user clicks through, that session is trackable. Correlate AutoRAG crawls with downstream visits to understand your AI search performance.
Find citation gaps. If AutoRAG frequently fetches certain pages but users never click through, your content may be answering the query completely in the AI response — leaving no reason to visit your site. That's a content strategy signal.
Spot 4XX and 5XX errors early. If AutoRAG encounters broken or unavailable pages, your content won't appear in AI-generated answers. Catching these errors in your logs lets you fix them before you lose visibility in AI search results.
From crawl to conversion: how AutoRAG affects your revenue
The bot fetches your content to use in AI-generated search results.
Cloudflare's AI search shows your page as a source in its answer.
A user reading the AI answer clicks your link and visits your site.
The visitor signs up, buys, subscribes, or engages.
Most tools stop at step 2 — they test prompts to see if your brand appears in AI answers. But appearing in AI answers doesn't mean you're getting traffic. The AI might summarize your content so completely that nobody clicks through.
The only way to measure the full pipeline is by connecting AutoRAG's crawl activity in your live traffic data with referral sessions in your analytics. That's exactly what Can AI See It does — including a Crawl-to-Referral Ratio (CRR) that shows how much traffic Cloudflare actually sends back for every 1,000 pages AutoRAG crawls.
What Can AI See It measures for AutoRAG
Pages fetched by AutoRAG per day, week, and month
Visits arriving from Cloudflare's AI search to your site
Your AI traffic efficiency score — referrals per 1,000 crawls
Revenue and goals attributed to AI-referred sessions
Without this data, you're making block/allow decisions blind. A bot with high crawl volume but zero referrals is just consuming your content. A bot with modest crawls but high CRR is driving real business value.
How is this different from prompt testing tools? Prompt testing checks if AI mentions your brand in simulated queries. Can AI See It measures what actually happens: real crawls, real referrals, real conversions — from your live traffic data.
Read: Why live traffic monitoring beats prompt testing →Log Verification
To verify AutoRAG traffic in your live traffic data:
- Search access logs for the user-agent strings listed above
- Check if the IP addresses match documented ranges (if provided by Cloudflare)
- Verify the crawl pattern matches documented behavior
- Use reverse DNS lookup for additional verification if available
Note: Observed behavior in production environments may differ from official documentation. Live traffic monitoring provides the only reliable verification of actual bot behavior.
Undocumented Information
The following information is not officially documented for AutoRAG:
- crawl frequency details
- IP verification method
- JavaScript rendering details
Live traffic monitoring vs prompt testing
| Prompt testing tools | Can AI See It | |
|---|---|---|
| What it measures | Does AI mention your brand in test prompts? | Does AI bot activity lead to real visits and conversions? |
| Data source | Simulated AI queries | Your live traffic data + analytics |
| Covers which bots | ChatGPT, Perplexity, Gemini (chat interfaces only) | 500+ bots including crawlers, assistants, and training bots |
| Business metric | Brand mention frequency | Crawl-to-Referral Ratio, revenue attribution |
| Tells you | "AI mentions you 34% of the time for keyword X" | "AutoRAG crawled 8,200 pages and Cloudflare sent 290 visitors who generated $4,100 in revenue" |
Measure your Crawl-to-Referral Ratio for AutoRAG
See how much traffic Cloudflare actually sends back to your site relative to how much content AutoRAG takes.
- Connect AutoRAG crawls in your logs with referral sessions in analytics
- Calculate your CRR — the metric prompt testing tools can't provide
- Make data-driven block/allow decisions for every AI bot
Measure business impact from AutoRAG
The real question is: does AutoRAG activity translate into referrals, sessions, and conversions on your site?
- Crawls: pages fetched by AutoRAG
- Referral tracking: AutoRAG takes — measure what Cloudflare gives back. Track actual visits arriving from Cloudflare's products to your site.
- CRR: your Crawl-to-Referral Ratio — referrals per 1,000 crawls
Based on your live traffic data and analytics — not synthetic prompt tests.
Official Documentation
View Official AutoRAG Documentation →
Information sourced from official documentation. Content generated with AI assistance.