What is Meta-ExternalFetcher?
Direct Answer: Meta-ExternalFetcher is a crawler operated by Meta, receiving individual links at user's initiative to support product features.
The Meta-ExternalFetcher crawler is operated by Meta and is used to receive individual links at the user's initiative. This supports certain product features. The crawler's user-agent string is 'meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)'. It does not follow robots.txt instructions from the database.
User-Agent Identification
The following user-agent strings identify Meta-ExternalFetcher in your live traffic data:
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
robots.txt Rules for Meta-ExternalFetcher
Respects robots.txt: No
Robots.txt has limited effect on user-initiated bots
Meta-ExternalFetcher is triggered by user actions within Meta's products. While Meta states it respects robots.txt, the bot operates differently from autonomous crawlers — it fetches specific URLs on demand rather than systematically spidering your site. Server-log monitoring is the only reliable way to verify what actually happens.
Need continuous verification across 500+ bots? Can AI See It automates this.
Crawl Behavior
Frequency:On-Demand
Request Pattern:Receives Individual Links At User'S Initiative
Crawl Activity Index
Relative crawl activity for Meta-ExternalFetcher over the past 28 days. Higher values indicate increased crawling intensity compared to the period baseline.
View recent activity data (last 7 days)
| Date | Activity Index |
|---|---|
| Mar 28, 2026 | 25.1 |
| Mar 29, 2026 | 29.4 |
| Mar 30, 2026 | 13.5 |
| Mar 31, 2026 | 31.7 |
| Apr 1, 2026 | 14.4 |
| Apr 2, 2026 | 23.2 |
| Apr 3, 2026 | 34.0 |
Source: Cloudflare Radar
Why track Meta-ExternalFetcher traffic?
Measure what Meta gives back. Meta-ExternalFetcher fetches your pages for AI-assisted answers — but do users actually click through to your site? Track referral visits from Meta's AI assistant relative to how many pages it fetches.
Track real-time content usage by AI assistants. Meta-ExternalFetcher fetches your pages when a user explicitly asks Meta's AI assistant to look something up. Each request represents a real user seeking your content through an AI intermediary.
Attribute AI-driven sessions. When someone clicks a link provided by Meta's assistant, that visit can be tracked in your analytics. Connect Meta-ExternalFetcher crawls to referral sessions to measure the full path: AI fetch → user click → conversion.
Understand content authority signals. The pages Meta-ExternalFetcher fetches most often reveal what Meta's AI considers authoritative on your site.
Detect 4XX and 5XX errors before users do. When Meta-ExternalFetcher hits a broken or unavailable page, it can't serve your content to users asking about it. Catching these errors early in your logs prevents lost visibility in AI-assisted answers.
Every Meta-ExternalFetcher request = a real person looking for your content
Unlike autonomous crawlers, Meta-ExternalFetcher only fetches your page when a real user asks for it. Every request in your logs represents someone who specifically wanted information from your site through Meta's AI assistant.
"Find me information from [your domain]"
Logged in your server access logs with timestamp and URL
User sees your information with a citation link
Trackable as a referral session in your analytics
This makes Meta-ExternalFetcher the most directly measurable AI traffic source. Can AI See It connects these dots with the Crawl-to-Referral Ratio (CRR) — showing what percentage of AI-fetched requests translate into actual site visits and conversions.
What Can AI See It measures for Meta-ExternalFetcher
How many real users asked AI to fetch your content
What % of AI-fetched visitors actually land on your site
Which content AI users ask about most
Conversions from AI assistant referrals
Blocking Meta-ExternalFetcher means cutting off a channel where real users are actively seeking your content. Before deciding, measure how many of these requests convert.
How is this different from prompt testing tools? Prompt testing checks if AI mentions your brand in simulated queries. Can AI See It measures what actually happens: real crawls, real referrals, real conversions — from your live traffic data.
Read: Why live traffic monitoring beats prompt testing →Log Verification
To verify Meta-ExternalFetcher traffic in your live traffic data:
- Search access logs for the user-agent strings listed above
- Check if the IP addresses match documented ranges (if provided by Meta)
- Verify the crawl pattern matches documented behavior
- Use reverse DNS lookup for additional verification if available
Note: Observed behavior in production environments may differ from official documentation. Live traffic monitoring provides the only reliable verification of actual bot behavior.
Undocumented Information
The following information is not officially documented for Meta-ExternalFetcher:
- crawl frequency details
- IP verification method
- JavaScript rendering details
Live traffic monitoring vs prompt testing
| Prompt testing tools | Can AI See It | |
|---|---|---|
| What it measures | Does AI mention your brand in test prompts? | Does AI bot activity lead to real visits and conversions? |
| Data source | Simulated AI queries | Your live traffic data + analytics |
| Covers which bots | ChatGPT, Perplexity, Gemini (chat interfaces only) | 500+ bots including crawlers, assistants, and training bots |
| Business metric | Brand mention frequency | Crawl-to-Referral Ratio, revenue attribution |
| Tells you | "AI mentions you 34% of the time for keyword X" | "Meta-ExternalFetcher crawled 8,200 pages and Meta sent 290 visitors who generated $4,100 in revenue" |
Measure your Crawl-to-Referral Ratio for Meta-ExternalFetcher
See how much traffic Meta actually sends back to your site relative to how much content Meta-ExternalFetcher takes.
- Connect Meta-ExternalFetcher crawls in your logs with referral sessions in analytics
- Calculate your CRR — the metric prompt testing tools can't provide
- Make data-driven block/allow decisions for every AI bot
Measure business impact from Meta-ExternalFetcher
The real question is: does Meta-ExternalFetcher activity translate into referrals, sessions, and conversions on your site?
- Crawls: pages fetched by Meta-ExternalFetcher
- Referral tracking: Meta-ExternalFetcher takes — measure what Meta gives back. Track actual visits arriving from Meta's products to your site.
- CRR: your Crawl-to-Referral Ratio — referrals per 1,000 crawls
Based on your live traffic data and analytics — not synthetic prompt tests.
Official Documentation
View Official Meta-ExternalFetcher Documentation →
Information sourced from official documentation. Content generated with AI assistance.