AI-Ready Bot Directory

Amazon Kendra is an intelligent search service operated by Amazon, using natural language processing for accurate search results.

Amazonbot is Amazon's web crawler used to improve products and services, such as training AI models.

The Anchor Browser is a web browser for AI agents, operated by Anchor, used for automating workflows and web interactions.

AwarioSmartBot is a web crawler operated by Awario, used for collecting new and updated web data for Internet marketers.

Big Sur AI Crawler, operated by Big Sur AI, crawls user websites for AI-infused experiences.

Brandwatch's Magpie Crawler indexes web pages for social media monitoring and analysis.

Ceramic TerraCotta is a web crawler operated by Ceramic, an AI training infrastructure company. It crawls websites to support Ceramic's AI model training optimization platform.

ClaudeBot is Anthropic's web crawler that collects web content for training Claude AI models. It respects robots.txt directives and supports Crawl-delay.

Cotoyogi is a bot operated by the Research Organization of Information and Systems for AI training purposes.

Echobot is a web scraping bot operated by Echobox for AI training purposes, specifically for automating content distribution for digital publishers.

Factset Spyderbot is a web scraping bot operated by Factset for delivering reliable financial data.

Google NotebookLM bot operated by Google for AI training.

Google-CloudVertexBot is a crawler operated by Google for targeted AI training of site owners' own sites.

GoogleOther is a generic crawler operated by Google for fetching publicly accessible content from sites, used for internal research and development.

GPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models.

ICC Crawler is a web crawler operated by NICT, collecting web pages for AI training.

LINER Bot is a web crawler operated by Liner Bot, used for AI training by collecting data from the internet.

Meta-ExternalAgent is a bot operated by Meta for AI training purposes, specifically for training AI models or improving products by indexing content directly.

The NetEstate Imprint crawler crawls websites for public contact information.

Novellum.ai is building out tools for building agents. This MCP tool will be used by agents to crawl sites.

PetalBot is a search engine crawler operated by Huawei, used for indexing websites and providing content recommendations in Petal Search engine, Huawei Assistant, and AI Search services.

QualifiedBot is a crawler operated by Qualified.com, Inc. to power AI products by crawling customer websites.

SemrushBot-OCOB is a bot operated by Semrush for ai-training, specifically for Content Toolkit.

SEMrushBotSwa is a bot operated by SEMrush for ai-training purposes, collecting data for the SEO Writing Assistant tool.

Not officially documented

WARDBot is a monitoring bot operated by WEBSPARK that tracks URL status codes for users.

Applebot data is used to power various features, such as the search technology that is integrated into many user experiences in Apple’s ecosystem including Spotlight, Siri, and Safari.

AutoRAG is Cloudflare's AI search solution for indexing and querying data.

Bravebot is a search crawler operated by Brave Software, Inc., used for discovering new pages and indexing their content.

OAI-SearchBot is used to link to and surface websites in search results in OpenAI's SearchGPT and ChatGPT search features.

Buy For Me agent places orders on e-commerce websites at the direction of customers.

ChatGPT Agent is OpenAI's agent mode that uses a visual browser to navigate websites and perform tasks on behalf of users.

ChatGPT-User is for user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a Custom GPT a question, it may visit a web page to help answer.

Devin is an autonomous AI software engineer operated by Devin AI, designed to assist engineering teams with coding tasks.

DuckAssistBot is a web crawler operated by DuckDuckGo for AI-assisted search answers.

Meta-ExternalFetcher is a crawler operated by Meta, receiving individual links at user's initiative to support product features.

MistralAI-User bot, operated by Mistral AI, for user actions in le Chat.

TwinAgent is a bot operated by Twin that automates complex operations end-to-end using AI.

Indexes content for Algolia search engine

All Africa Crawler, operated by AllAfrica Global Media, aggregates news from Africa.

Baidu's DDoS scrubbing proxy.

Baiduspider is the search engine crawler for the search engine Baidu.

Bingbot is a crawler from Microsoft that powers Bing search.

Cốc Cốc bot, operated by Coccoc, is a search engine bot that scrapes websites requested by the Vietnamese search engine Coc Coc.

Coveo Bot is a search engine bot operated by Coveo, providing relevant experiences to end users by crawling websites.

Crawlson is a search engine crawler for the crawlson.com search engine.

Dataprovider.com indexes the web and structures the data.

Daum is a Korean search engine crawler operated by Daum.

DuckDuckBot is a web crawler for DuckDuckGo. DuckDuckBot's job is to constantly crawl the web and find web pages to add to the DuckDuckGo search index.

Freespoke is a search engine that believes in free speech and shows you all viewpoints.

Funnelback is an enterprise search platform operated by Squiz, used for indexing content from an organization's websites and data repositories to power internal search functions.

Gigablast is the only non-Big Tech search engine in the U.S. that uses its own search index and algorithms.

Google Images bot is a search engine crawler operated by Google for Google Images Search.

Google Scholar bot crawls and indexes scholarly literature from academic publishers, repositories, and university websites.

The Google Videos bot is a search engine crawler operated by Google for Google Video Search.

Google uses crawlers and fetchers to perform actions for its products, either automatically or triggered by user request.

Hype Machine is a search engine bot operated by Hype Machine that monitors music publications/blogs for new artists.

IbouBot is a search engine crawler operated by Babbar, used to fuel and update the graph representation of the World Wide Web.

jobswithgptcom-bot is a crawler operated by jobswithgptcom, focusing on job postings for jobswithgpt.com.

Kagi Bot is a web crawler for the Kagi search engine, operated by Kagi, used to build its own search index for its ad-free search product.

Level9SearchBot is a search engine bot operated by Level 9.

Linespider is a web crawler operated by LINE Corporation, providing search results for LINE services.

MagiBot is owned by Peak Labs which focuses on the research and development of information extraction and retrieval technology to transform knowledge in natural language into immeasurable value.

Marginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs.

Mars Finder is a website search service operated by Mars Flag, designed to utilize the maximum potential of a website.

MojeekBot is a web crawler for the Mojeek search engine, operated by Mojeek.

MRGbot is a search engine bot operated by MRG Web Services srl, aimed at generating a corpus of data.

MSNBot is a web crawler operated by Microsoft for indexing web pages for MSN Search.

Neevabot is the web crawler for the search engine neeva.com.

NewsNow bot is a web crawler for the news aggregator service NewsNow, operated by NewsNowUK.

Notabot is a crawler operated by Helpfeel for integrating external search engine.

The Potions bot, operated by Potions, fetches product feeds and crawls e-commerce data.

Qwantbot is a web crawler operated by Qwant, a European search engine that prioritizes user privacy, used for search indexing and web content discovery.

Seekport is a search engine bot operated by SISTRIX, providing a public, free, and independent alternative to Google.

SeznamBot is a search engine crawler operated by Seznam for search indexing.

SiteSearch360 is a search engine bot operated by SEMKNOX GmbH, used for indexing content on customer sites for search.

SMTnet PM Bot is a search engine crawler operated by SMTnet, crawling partner company websites for on-site search engine inclusion.

Sogou spider is a search engine crawler operated by Sogou, used for indexing web content.

Toutiao is ByteDance's automated news aggregation bot collecting content across web platforms.

Yahoo Slurp is a search engine crawler operated by Yahoo, used for indexing web pages and collecting content.

Yahoo! JAPAN manages and operates a system that accesses web pages published on the Internet for the purpose of providing services, research, development, maintenance, etc.

The main indexing robot for Yandex search.

Yeti is a web crawler operated by Naver, a South Korean search engine, used for indexing websites to provide search results.

Zoominfobot is an indexing robot for a web search engine, similar to Google. Created by Zoom Information Inc.(www.zoominfo.com), Zoominfobot’s patented technology continually scans millions of corporate websites, press releases, electronic news services, SEC filings and other online sources. Usin...

ZumBot is a web crawler operated by Zum Internet Corp for indexing webpages for Zum Open Internet Search.

AddSearchBot is a search indexing bot operated by Addsearch, used to add content from sites to their search results.

The AddThis bot, operated by AddThis, crawls websites to gather and update content for its website marketing tools.

Agency Analytics Crawler is a web crawler operated by Agency Analytics for SEO site checking.

Ahrefs Site Audit bot, operated by Ahrefs, used for SEO site auditing.

AhrefsBot is a web crawler operated by Ahrefs for SEO toolset, crawling the web to fill its 12 trillion link database.

The Attracta bot analyzes website content for SEO services, operated by Attracta.

Barkrowler is an SEO web crawler operated by Babbar, used to identify web page popularity and thematic.

BLexBot is a backlink analysis tool operated by WebMeUp, providing comprehensive backlink data for SEO purposes.

Botify is an SEO tool operated by Botify, used for website crawling and analytics.

CaliberBot is a web crawler operated by Conductor for SEO tool purposes, extracting HTML features for Content Analytics.

Chrome-Lighthouse is an open-source tool by Google for auditing web page quality, not a traditional web crawler.

ClearscopeBot is a web scraper operated by Mushi Labs for SEO content optimization, retrieving publicly available content from websites.

Cloulflare internal service that crawls customer error pages in order to serve them directly from our edge network.

Cloudflare Digicert DCV service operated by Cloudflare for domain control validation.

Cloudflare Healthchecks service

Cloudflare Prefetch is a bot operated by Cloudflare for URL prefetching to improve cache hit rates and user experience.

Cloudflare Purge service.

Cloudflare SpeedTest service operated by Cloudflare for SEO tool purposes.

Cloudflare SSLDetector service.

Cloudflare Stream Webhook service.

Cloudflare-Traffic-Manager service.

CognitiveSEO is an SEO toolset that crawls the web and analyzes links.

Cxensebot is a bot operated by Cxense for SEO monitoring and analysis.

DataForSEO bot operated by DataForSEO for SEO and digital marketing data solutions.

DataForSEO Bot is an SEO tool operated by DataForSEO, used for backlink analysis and providing website owners, webmasters, and SEO professionals with backlink analytics.

DigiCert DCV bot operated by DigiCert for SEO tool purposes.

EzoicBot is a web crawler by Ezoic Inc, designed to extract information about internet, search engines, and websites to help publishers understand site performance and search engine ranking.

The Digital Asset Links bot verifies statements lists made by website operators.

Google StoreBot is a search-engine-based program that automatically crawls web pages to gather and analyze data, operated by Google.

Google-AdWords-Express is a Google bot for AdWords product aimed at small businesses, crawling advertiser websites for ad creation and site verification.

Google-Adwords-Instant is a Google bot that visits advertiser landing pages for verification and quality checks.

Huckabuy Bot is a web crawler operated by Huckabuy for SEO tool purposes, crawling and indexing websites for optimization.

Siteimprove's LinkCheck crawler analyzes websites for quality assurance, SEO, and accessibility purposes.

LinkChecker Bot is a web crawler operated by LinkChecker that monitors backlinks profiles.

marketgoo is an SEO tool operated by marketgoo, providing white label SEO tools.

MJ12bot is a web crawler operated by Majestic for SEO backlink analysis and search engine data mapping.

Moz Dotbot is a web crawler operated by Moz for SEO backlink analysis and site auditing.

Moz's site audit crawler for Moz Pro Campaigns, operated by Moz.

NitroBot is a site speed optimization bot operated by NitroPack Ltd, making requests to clients' sites to fetch content for optimization.

OnCrawl is an enterprise SEO platform powered by an industry-leading SEO Crawler and Log Analyzer, operated by Oncrawl.

Online Webceo Bot is a bot from Online Webceo that scans websites for SEO analysis and technical audits.

It's HTML pre rendering service for SPA(Single Page Application) Website SEO.

Our bot crawls our customers' websites to identify SEO opportunities

SearchAtlas Bot is an SEO tool operated by Search Atlas, used to evaluate customer websites and provide SEO optimization strategies.

SemrushBot is the search bot software that Semrush sends out to discover and collect data from the web for SEO analysis and competitive research.

SEMrushBotBacklinkAudit is a bot operated by SEMrush for SEO backlink analysis and site auditing.

SemrushBotBacklinks is a bot operated by Semrush for collecting data for Semrush's Backlink Analytics tool.

SEMrushBotLinkBuilding is a bot operated by SEMrush for SEO backlink analysis.

SEMrushBotSI is a bot operated by Semrush for SEO tool purposes, specifically for data collection.

SEO Audit Check Bot is an automated tool by SEO Audit Check that performs comprehensive SEO audits on websites.

The seo4ajax bot, operated by Capsule Code, helps make single-page applications (SPAs) crawlable by search engines.

The Seobility Bot is an SEO tool operated by Seobility that crawls websites to gather SEO information.

SerpstatBot is a bot operated by Serpstat, used for SEO backlink analysis and site auditing by collecting data for Serpstat's Backlink Analysis tool.

SiteAuditBot is a SEO tool operated by Semrush that checks for common website issues and provides reports on crawlability, markups, internal linking, speed, HTTPS, and international SEO.

Siteimprove Crawl bot, operated by Siteimprove, for SEO and content suite analysis.

Trellis-Services bot, operated by Mediavine, generates critical CSS for website optimization.

W3C Validator Services is a bot operated by World Wide Web Consortium (W3C) that checks website conformance to open standards.

SEO crawler for Yahoo Japan.

Bing Preview is a bot operated by Microsoft that generates page snapshots for Bing.

Bluesky social pulls links in advance to render webpage previews.

Buffer Link Preview Bot is a web crawler operated by Buffer, a social media management platform, that generates rich previews when users share links.

The discordBot, operated by Discord, Inc., scrapes URLs shared within the Discord chat platform to generate contextual previews.

Facebook External Hit crawler scrapes content from the web to generate link previews when URLs are shared on Facebook, Messenger, and other Meta platforms.

Fastmail bot for image proxying and calendar fetching, operated by Fastmail.

Fedicabot is a social media publishing helper bot operated by Fedica, reading user-specified pages and RSS feeds.

We work with publishers/partners to obtain their content so it is formatted for the Flipboard app.

The Google Image Proxy bot, operated by Google, renders link content sent via email in Gmail.

Google Page Renderer is a user-triggered fetcher operated by Google for rendering web pages.

Google Web Snippet is a bot operated by Google for generating web snippets, running on Linux.

HubSpot Page Fetcher is a bot operated by HubSpot, used for fetching pages, specifically for posting to LinkedIn with images.

Iframely is a rich media embed solution operated by Iframely.

LinkedInBot is a social media bot operated by LinkedIn for rendering link preview information.

LinksIndexerBot is an SEO bot operated by Links Indexer that crawls websites to index backlinks and aggregate website summaries.

Loomly Bot is a social media bot operated by Loomly, used to extract metadata for social media post previews.

MailRUBot is a bot operated by Mail Russia, used for fetching mail on behalf of the Mail.ru email service.

Meta-WebIndexer is a social media web crawler operated by Meta for search indexing and web content discovery.

Microsoft Preview bot generates page snapshots for Microsoft products.

Opengraph Bot is a social media bot operated by Opengraph, used for previewing links when sharing on platforms.

The Orlo Link Preview bot is used by the Orlo social media management platform to fetch previews of links scheduled to be published in social media posts.

Ozon Web Grabber is a bot operated by Ozon, used for loading previews of external and internal links.

The PressEngine Bot verifies video game press coverage, operated by PressEngine.

Shortwave Image Fetcher bot, operated by Shortwave Communications Inc., proxies images in HTML emails to protect user IP addresses and connections.

Skype's URI Preview services fetches a page preview when someone posts a URL in a Skype message.

Slack Image Proxy bot, operated by Slack, fetches and caches images posted in Slack channels.

The Steam Chat bot, operated by Valve Software, fetches URL previews shared in Steam's chat feature.

TelegramBot crawls websites to render a link preview when people send a message containing a URL in the Telegram messaging service.

Tumblr bot operated by Automattic, unfurls URLs into Link Blocks for posts.

Twitterbot is a Twitter bot operated by Twitter that autonomously performs actions via the Twitter API.

WordCountBot analyzes website word count based on public pages. All words belonging to public pages and included in HTML source code

Yahoo Link Preview is a bot operated by Yahoo that fetches data from URLs shared on Yahoo platforms to provide thumbnail images, page titles, and descriptions.

Yahoo Mail proxy bot operated by Yahoo, fetches link content rendered in Yahoo's webmail service.

YahooCacheSystem is a bot operated by Yahoo that caches website contents for the Yahoo! Search Service.

Yahoo Mail Proxy is a content fetch proxy operated by Yahoo that retrieves page content of URLs embedded in emails sent to Yahoo Mail users.

360Monitoring bot checks website availability and performance from selected locations.

AccessStatus bot checks HTTP status codes of web pages to ensure link functionality.

ActiveComply Bot is a crawler for social media monitoring and regulatory compliance, operated by ActiveComply.

Adagio Bot, operated by Adagio, for monitoring and demand optimization solutions.

Alertsite by Smartbear is an HTTP monitoring probe operated by Smartbear that monitors websites for availability and performance anomalies.

Area 360 is a property search and analytics bot operated by Area360, used for monitoring property insights.

Audisto Crawler is a website monitoring bot operated by Audisto, fetching all accessible URLs of a website.

Automaton is an end-to-end campaign and integration testing tool operated by Automaton, used for monitoring and optimizing marketing, advertising, and sales technology stacks.

Better Uptime is a monitoring bot operated by Better Stack, used for monitoring infrastructure platforms.

BinaryCanary is a website monitoring bot operated by Binary Canary, tracking availability and performance issues.

BlogVault is a monitoring bot operated by BlogVault, providing WordPress services like backup, security, and monitoring.

BugsNag is a monitoring bot operated by Bugs Nag for error and performance monitoring of web applications.

Catchpoint is a monitoring bot operated by Catchpoint, focused on internet performance monitoring from the end-user perspective.

Checkly Bot is a monitoring bot operated by Checkly, used for proactive monitoring and testing of websites and APIs.

Cloudflare diagnostic and testing services.

Cloudtrellis bot scans websites for broken links, accessibility issues, and SEO improvements, operated by Cloudtrellis.

Cludo is a monitoring bot operated by Cludo, facilitating search and analytics solutions.

ContentKing is a cloud-based service that monitors websites for technical issues and content changes.

Crazy Egg bot that takes screenshots of pages, collects assets, tests script installation.

Dead Link Checker bot, operated by DLC Websites, crawls websites to identify broken links.

Deepcrawl bot, operated by Lumar, crawls websites for performance analytics and SEO optimizations.

The Doctom Monitor is a website monitoring bot operated by Dotcom-Monitor, checking websites for uptime and performance issues.

Dr. Link Check is a monitoring bot operated by Dr. Link Check that crawls websites to identify and fix broken links.

Easydns is a monitoring bot operated by EasyDNS, used for uptime monitoring.

elmah.io Uptime Monitoring bot monitors website availability for elmah.io users.

eRepublik.tools is a monitoring bot operated by Sebastian Foth - Software Solutions, providing gaming statistics and tools for the browser game 'eRepublik'.

EvoUptimeBot is a monitoring bot operated by Evo Agency Ltd. for uptime monitoring of the EvoCommerce platform.

FDL Stats Bot, operated by FTW Entertainment LLC, generates analytical data on rocket league player information.

Foregenix ThreatView/WebScan is a security and risk scanning bot operated by Foregenix Limited, monitoring websites for malware and vulnerabilities.

FullStory is your digital experience analytics platform for on-the-fly funnels, pixel-perfect replay, custom events, heat maps, advanced search, Dev Tools, and more. FullStoryBot’s fetches and stores assets required to rebuild sites when viewing recorded sessions.

Ghost Inspector is a monitoring bot operated by Ghost Inspector for automated browser testing.

Google Schema Markup Testing Tool bot, operated by Google, validates structured data on web pages.

Grafana's Synthetic Monitoring agent provides probe functionality and executes network checks for monitoring remote targets.

GTmetrix is a free tool that analyzes a page's speed performance. Using PageSpeed and YSlow, GTmetrix generates scores for pages and offers actionable recommendations on how to fix them.

GuestpostsBot is a Web Crawler that has several functions to facilitate the website owner who has registered his site on the guestposts.com.br platform to monitor his site. The bot constantly tracks the sites registered on the platform in order to check if the partnerships made on the guestpost p...

HetrixTools Uptime Monitoring Bot checks website and server uptime, operated by HetrixTools Inc.

HoneybadgerBot is a monitoring bot operated by Honeybadger Industries for error and uptime monitoring services.

HostTracker is a monitoring bot operated by HostTracker that tracks website availability and performance.

Hotjar is a user analytics and feedback tool operated by Hotjar, now part of Contentsquare.

InternetArchiveBot is a monitoring bot operated by Internet Archive that checks URL references on Wikipedia for broken links.

IsDownBot monitors endpoints to ensure they are up and running, operated by IsDownBot.

UptimeBot by Jagged Pixel Inc. monitors Shopify store uptime and alerts for outages.

Jetpack is a monitoring bot operated by Automattic, tracking uptime for WordPress.com and Jetpack users.

KargoBot-Artemis is Kargo's autonomous content verification bot. It's a simulation of a user on an iOS device. The bot is used to scan sites for content that may be unsuitable for customers on the Kargo ad network.

Kinsta bot for uptime monitoring and domain pointing verification

LinkTiger is a monitoring bot operated by LinkTiger, Inc. that crawls customer sites and reports broken links.

LogicMonitor bot performs synthetic monitoring checks on websites and online services.

Macrobondbot is a custom web crawler operated by Macrobondbot to fetch macroeconomic data from official sources.

The Mavifinds Bot is part of a security service that monitors websites. It can automatically activate security measures, such as an under attack mode, in response to threats.

Medialogia Bot is a web crawler operated by Medialogia, a Russian media monitoring company, collecting data from online news and social media.

Monitage is a monitoring bot operated by Monitage, used for uptime monitoring of websites and services.

Monitis bot for premium monitoring service, operated by Monitis.

Monsidobot is a monitoring bot operated by Acquia, used for scanning and improving content and web accessibility.

MonSpark is a website monitoring service operated by MonSpark, checking website availability, network conditions, and TLS certificate validity.

MontasticMonitor is a website availability monitoring bot operated by Montastic.

Netumo is a website uptime monitor service operated by Netumo, providing monitoring and verification of online infrastructure.

NETVIGIE is a monitoring bot operated by Netvigie, checking website and mobile app availability and performance.

New Relic bot, operated by New Relic, monitors customer applications for availability and performance issues.

NixStatsMonitoringBot is a monitoring bot operated by NixStats to track website availability and performance.

Nodeping is a monitoring bot operated by Nodeping that checks websites for uptime.

Oh Dear is a monitoring bot operated by Oh Dear, used for checking website availability and performance.

The Onetrust CMP Scanner is a monitoring bot operated by OneTrust, LLC, that identifies and categorizes cookies and tracking tech on customer sites.

OnlineOrNot Monitor is a website monitoring bot operated by OnlineOrNot, providing uptime checks and page speed tests.

OutsellURLValidator bot by Outsell Corporation monitors client website URLs for hosting/provider changes.

PaesslerCloudBot is a monitoring bot operated by Paessler, used for website availability and performance monitoring.

Panopta is an uptime monitoring service acquired by Fortinet.

Parse.ly is a content analytics tool operated by Parse.ly, used for monitoring content performance.

Pingdom bot is an HTTP monitoring probe for Pingdom's website monitoring service, operated by Pingdom.

PingPing is a website monitoring service operated by PingPing that checks website uptime and TLS certificate validity.

PingPing Bot is a monitoring bot operated by PingPing.io to check website uptime and SSL validity.

Pulsetic is a website monitoring service operated by Pulsetic, providing uptime monitoring and down alerts.

Qualys SSL Scanner is a monitoring bot operated by Qualys, used to test and monitor SSL ratings.

The Rackspace HTTP Monitor monitors customer websites for uptime and other issues.

RetroListeCOM is a monitoring bot operated by Niclas Papst, tracking user counts on gaming-related websites.

Crawler for analyzing SE Ranking clients websites for potential issues.

Sentry Uptime Monitoring bot checks websites and APIs for availability, operated by Sentry Uptime Bot.

SequelWP monitoring bot checks website uptime every 5 minutes for verified customers.

Server Density bot by Stackpath for proactive infrastructure monitoring

Site24x7's global website monitoring probe operated by Site24x7 for monitoring purposes.

SiteUpTimeBot is the HTTP monitoring probe for SiteUpTime.com.

SolarWinds Observability is a monitoring bot operated by SolarWinds, used for synthetic monitoring of websites and services.

Spectate is an uptime monitoring bot operated by Spectate, used for healthchecks.

Splunk is a monitoring platform operated by Splunk, used by Enterprise organizations for synthetics monitoring.

Synthetic monitoring tool with global agents running in AWS

Stape Scanner is a monitoring bot operated by Stape Inc, used for evaluating tracking tag configurations.

StatusCake is a monitoring bot operated by StatusCake, used to monitor webpage availability and performance.

Swifteq Link Checker bot by Swifteq Ltd for monitoring and detecting broken links in Zendesk help centers.

Datadog Synthetic Bot, operated by DataDog, for monitoring applications and API endpoints via simulated user requests.

TermlyBot is a web crawler operated by Termly for detecting and categorizing website cookies.

ThousandEyes monitors network infrastructure, troubleshoots application delivery and maps Internet performance, all from a SaaS-based platform.

Trendiction Bot is a monitoring bot operated by Trendiction S.A.

UpDownBot is the HTTP monitoring probe for updown.io, operated by Updown, for monitoring website uptime and performance.

Uptime LLC bot, operated by Uptime, for website availability and performance monitoring.

UptimeBot is an actionable website monitoring tool that works great with Slack.

UptimeDoctor is a website uptime monitoring service.

UptimeRobot is a free website uptime monitoring service that checks if your website is up or down at configurable intervals.

Uptimia is a website monitoring service operated by Uptimia, monitoring website performance and availability.

UptrendsBot is a monitoring bot operated by Uptrends GmbH for synthetic monitoring and load testing.

Watchbot is a monitoring bot operated by WatchBot, tracking website availability and SSL certificate expiration.

Watchful is a monitoring system operated by Watchful LLC to check uptime on client websites.

WebPageTest is a web performance testing tool operated by Catchpoint, used for measuring webpage performance from various locations and browsers.

WebsitePulseBot is an HTTP monitoring probe operated by WebsitePulse for their monitoring service.

WkndHealthCheckBot is a monitoring bot operated by Wunderkind, used for performing health checks on clients' domains.

WormlyBot is an HTTP monitoring probe operated by Wormly for uptime monitoring.

WP Umbrella bot fetches data from WordPress sites for Umbrella plugin users, operated by WP Umbrella.

WPMU DEV Broken Link Checker is a bot operated by WPMUDEV that scans sites for broken links.

WPMUDEV Uptime Monitor 5.0 is a website monitoring bot operated by WPMUDEV, checking site uptime and performance.

Arquivo Web Crawler is a web archiving bot operated by Arquivo, capturing the Portuguese web.

Internet Archive's Archive-It bot preserves web pages for historical records.

The Internet Archive bot crawls and preserves publicly accessible web pages for the Internet Archive's Wayback Machine historical record.

Google AdsBot is a special-case crawler that monitors websites where Google Ads are served, checking landing page quality and ad compliance.

AdsTxtCrawler by OneTag Limited crawls ads.txt files to verify onetag.com line presence.

Amazon AdBot is a crawler used by Amazon's advertising services to ensure advertiser landing pages comply with advertising policies.

Amazon Contxbot is a crawler bot operated by Amazon for advertising purposes, visiting sites to determine content for relevant ads.

BigUpData Bot, operated by BigUpData, automates product deals pipeline to advertisement.

Bing Ads bot operated by Microsoft for ad quality control.

Autopilot is an SEO marketing automation tool that includes features for internal linking and image optimization. We crawl customer sites so that we can determine the best links to use on the site and to find images that need to be optimized.

Channable is a tool used by some Cloudflare customers. The tool downloads data from the customers' systems for marketing automation and e-commerce automation.

The Clickagy Intelligence Bot is an ad verification bot for Clickagy.

ContextualBot is a web indexer operated by Outcomes that analyzes webpage content to create contextual segments for targeting within advertising platforms.

CriteoBot is a web crawler operated by Criteo for serving relevant ads.

The Mediapartners-Google bot, operated by Google, monitors website content for AdSense.

Google Firebase is a crawler operated by Google for advertising purposes.

Google-Display-Ads-Bot is a crawler operated by Google to verify sites when added to AdSense.

The Grapeshot bot, part of Oracle Advertising, crawles web pages for content analysis and classification.

Groovinads bot gathers e-commerce data for ad services, operated by Groovinads.

IAS crawler is a bot operated by Integral Ad Science for digital ad verification and content rating.

Klaviyo is a B2C CRM bot operated by Klaviyo, used for advertising and marketing automation.

LeikiBot, operated by DoubleVerify, crawls webpages for advertisers.

Our App is called Grow and we allow publishers to enable bookmarking, social sharing, and searching on their sites.

Meta-ExternalAds is a web crawler operated by Meta, used for improving advertising and business-related products and services.

MgidBot is a bot operated by MGID for detecting context categories of content for advertising recommendations.

Missinglettr Bot crawls blog posts to turn them into social media campaigns.

Noorobot is an SEO tool operated by Noor Digital Agency AB that crawls websites to provide recommendations and identify SEO-impacting problems.

NostoCrawlerBot is a bot operated by Nosto for advertising purposes, used for crawling and accessing web pages.

Outbrain bot, operated by Outbrain, analyzes content for ad serving.

Proximic bot, operated by ComScore, crawls websites to determine ad suitability.

Quantcastbot is a web crawler used by Quantcast for advertisement quality assurance and to understand page content for Interest-Based Audiences.

Reelevant is an advertising bot operated by Reelevant, allowing customers to dynamically update content inside their emails by fetching images of their product at runtime.

Salesforce bot operated by Salesforce for advertising purposes with CRM and marketing capabilities.

SalesViewerBot is a web crawler operated by SalesViewer GmbH for advertising and sales intelligence purposes.

Slickstream is a SaaS bot operated by Slickstream, providing engagement features like site search and content recommendations.

SmartologyBot is a bot operated by Smartology, used for generating semantic vectors from domain pages to serve semantically-relevant ads.

Taboola bot, operated by Taboola, visits websites to audit content and gather metadata for advertiser campaigns.

The Trade Desk Content bot, operated by The Trade Desk, classifies webpage content for ad placement.

Yahoo Ad Monitoring bot, operated by Yahoo, monitors webpages for Yahoo ads.

Yahoo Japan Advertising Bot operated by Yahoo Japan Corporation for advertising purposes.

The Yext Crawler is a bot operated by Yext, used for retrieving data from websites for Yext customers.

Black Duck Fast Dynamic is a security scanning bot operated by Black Duck Software, performing Fast Dynamic DAST security scanning.

Scanning the internet to find malicious sites that scam crypto users into draining their wallets

Bushbaby is a security bot operated by Cloudflare to manage and renew SSL certificates.

Cert Chief is a security bot operated by Chief Tools that monitors HTTPS domains for certificate issues and configuration problems.

Cloudflare Radar URL Scanner scans and analyzes URLs for security threats, phishing, and malware on behalf of Cloudflare's security services.

Cookie Hub bot detects cookies set by websites for CookieHub clients and verifies user consent.

Within the GDPR legislation it is mandatory to ask a visitor for permission before placing so-called marketing or tracking cookies. Many websites contain a cookie notice, but what is not clear to everyone is that those cookies may not be placed before the visitor has given explicit permission. Co...

Cookiebot scans websites for cookies and trackers, operated by Cybot A/S for security purposes.

CookieYesbot is a security and compliance scanner operated by CookieYes to verify cookie consent banner implementations and compliance configurations on websites.

Detectify is a security bot operated by Detectify that analyzes web application security levels.

The Drata Autopilot bot continuously monitors the security posture of customer domains.

Google Inspection Tool is a crawler used by Google for Search testing tools, operated by Google.

Google Safety bot crawls for abuse-specific purposes like malware discovery on Google properties.

Google Site Verification bot verifies site ownership for Google Search Console, operated by Google for security purposes.

Google Trust Services bot for DCV Check, operated by Google, for certificate validation.

Digital ID Verification and Scanning Tool used by Coinbase team.

Monitor customers' sites for compliance with cookie- and other privacy-related legislation

Let's Encrypt bot for domain control validation, operated by Let's Encrypt.

Netcraft bot analyzes web server technology stacks for their Web Server Survey.

We use Project Honeypot for IP info.

ProjectShield Url Check bot operated by Google for security purposes, providing DDoS attack protection.

Qualys bot, operated by Qualys, crawls and tests web applications for vulnerabilities.

With the iboss Cloud Platform, each customer gets dedicated source cloud IP Addresses which are associated with the organization. Because of this, any data traversing the global cloud containerized gateways in the Platform will have a uniquely associated IP Address that can be mapped to the organ...

Sansec Security Monitor is a web crawler that enhances e-commerce security by monitoring stores for malicious code, preventing data breaches and digital skimming.

Provides a free security scanning service at https://securityheaders.com

Sentry is a security bot operated by Sentry that monitors webpages for availability and performance issues.

SiteLock is a security bot operated by SiteLock that scans websites for malware and malicious code.

SmarshBot is a security bot operated by Smarsh, providing archiving solutions for compliance purposes.

Splunk Attack Analyzer is a security bot operated by Splunk, analyzing URLs for malicious content using a headless Chrome browser.

The Sucuri bot is a security bot operated by Sucuri, crawling websites to scan for malware, security risks, and blacklisting status.

WebTotemBot is a security monitoring and defense bot operated by WebTotem, protecting web applications and their data.

2Checkout Payment Platform's webhooks integration bot, operated by Verifone, for notifying websites about payment events.

Accessible Web Bot crawls websites to discover pages and monitor for accessibility violations.

Adyen bot operated by Adyen, sends HTTP requests for payment-related events.

AfterShip Magento 2 Connector bot operated by AfterShip for order tracking and delivery updates.

Akismet is a spam filtering service operated by Akismet, protecting websites from comment, form, and text spam.

The Apple App Site Association bot, operated by Apple, supports Universal Links in native iOS apps by requesting specific paths for given hostnames.

Artemis is a calm, independently-run, free web reader. The reader often runs into Cloudflare access blocks which makes it hard to follow many sites.

The Authory bot visits websites to back up articles on behalf of journalists and other writers who use the service.

Bazqux Fetcher is a user-initiated bot operated by Bazqux that collects and refreshes RSS/Atom feeds and comments for users subscribed to BazQux Reader.

BestChange Bot operated by BestChange, downloads exchange rate information from 600 websites every 5 seconds.

The Bibliotheque Nacional de France Crawler is a bot operated by the Bibliothèque nationale de France, tasked with capturing and preserving the national documentary heritage.

Bitbucket bot operated by Atlassian for CI/CD and webhooks.

Bling ERP is an online ERP system operated by Bling that integrates with ecommerce platforms.

Blogtrottr is a service operated by Blogtrottr that delivers updates from news, feeds, and blogs to users' email inboxes.

BoardGamePrices Bot is a price comparison bot operated by KP Software Consult ApS, crawling store pages for board game prices.

Buttondown RSS-Feed-Parser bot fetches RSS feeds for Buttondown newsletters.

Capital One Bot crawls dealer websites for usage information.

Channel3Bot is a web crawler operated by Channel3 that indexes product pages to drive traffic back to those websites.

ChargeBee bot, operated by ChargeBee, for payment event notifications via webhooks.

Cloudflare CSUP is a bot used by Cloudflare's customer support for diagnostic purposes. It is not a general web crawler and is used to investigate technical issues with customer websites.

Cloudflare Validator is a bot operated by Cloudflare that verifies IPs for Cloudflare Bots Directory.

Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments.

ConvermaxBot is a site search indexer operated by Convermax Corp.

The Critical CSS Bot is operated by Critical CSS, a service for website optimizations that generates critical CSS to improve website performance.

cron-job.org is a service for scheduled execution of websites and scripts, operated by cron-job.org.

EasyBill Import Manager is a tool operated by easybill.de that synchronizes order data from external systems to EasyBill.

EasyCron is an online cron job service operated by EasyCron, allowing users to schedule HTTP requests.

eMoney Advisor bot collects financial data for planning and analysis, operated by eMoney Advisor.

Epivoz Crawler is a news aggregator bot operated by Techmeme, used to crawl news and blog articles to generate short summaries for page previews of attributed links.

Exodus bot, operated by Exodus, manages cryptocurrencies like Bitcoin, Ethereum, and Ripple.

FedReporter Bot for FFIEC, operated by Fed Reporter, Inc., downloads public banking institution data.

Feedbin is a web-based RSS reader operated by Feedbin.

Feeder is a cloud-based RSS reader operated by Really Simple AB, used for tracking online sources and bundling content into an easy-to-digest reading experience.

Feedly RSS fetcher service operated by Feedly, collecting and refreshing user-initiated feeds.

FeedWind Crawler by Mikle accesses feed sources to update the feed widget.

FlipboardRSS bot operated by Flipboard, uses visitor's RSS feed to discover articles and generate summaries.

Freshping is a website monitoring bot operated by freshping that checks if a website is online and sends alerts when it's down.

Google API bot for delivering push notification messages.

Google Feed Fetcher is a bot operated by Google for fetching RSS feeds.

Google Publisher Center bot operated by Google, fetches and processes publisher-supplied feeds for Google News landing pages.

Google Read Aloud is a service by Google that enables reading web pages using text-to-speech (TTS).

GoPay payment gateway http notification service operated by GoPay.cz

Hatena's service bot for automatic web page visits to gather information.

HelloWork bot aggregates job listings for the HelloWork platform, operated by HelloWork Group.

HIFIBot is a bot operated by HIFI, a financial services company for musicians and professional creators, used for automating retrieval and processing of royalty earnings statements.

The HSTS preload bot, operated by Chromium, checks that links can be preloaded for Chromium.

HubSpot Crawler is a bot operated by HubSpot for marketing, sales, and customer service platform.

HubSpot Feed Fetcher bot operated by HubSpot for pulling images when posting to LinkedIn.

IFTTT RSS Feed Service fetches RSS feeds to power user-configured automations.

Indeed's job crawling bot that crawls job and job related information

Innguma Fetcher is a user-controlled feed retriever operated by Innguma, collecting and periodically refreshing user-initiated RSS or Atom feeds.

Inoreader is an RSS feed reader operated by Innologica, used for content aggregation and discovery.

Instapaper is an app operated by Instant Paper that lets people save articles to read later.

Over 50 active Cloudflare users are using Integromat to automate their workflow.

The kb.dk_bot is operated by Netarkivet, a part of the Royal Danish Library, to collect and preserve the Danish Internet for research purposes.

The Library of Congress Web Archive is a bot operated by the United States Library of Congress that manages, preserves, and provides access to archived web content.

MagnetmeBot is a respectful web crawler operated by Magnet.me to discover and update career-related content.

MainWP bot for managing websites through the MainWP Control Dashboard.

Make is a no-code platform to help with task and workflow automation.

ManageWP is a web management tool operated by ManageWP, used to monitor and maintain multiple WordPress websites from a single dashboard.

MediaMonitoringBot is a web crawler operated by MediaMonitoringBot that indexes news and media publishers' websites.

Mediatoolkitbot is a media monitoring tool operated by Determ that crawls the internet for user-specified phrases.

Metorik is an analytics and email automation service bot operated by Metorik, used by eCommerce businesses to sync data from customer sites via API requests.

MirrorWebCrawler is a web archiving bot operated by MirrorWeb Ltd, providing archival solutions for financial and public sectors.

Modular DS is a service operated by Uniqoders Technologies SL to manage WordPress websites via the WP JSON API.

Mollie Bot operated by Mollie B.V., a payment service provider, uses webhooks to notify merchants about payment updates.

MonitoRSS is a bot operated by MonitoRSS that polls RSS feeds of Cloudflare-protected sites to deliver news articles to Discord servers.

MotoMinerBot is a web crawling bot operated by Motominer, used for indexing vehicle detail pages.

Muck Rack uses many approaches for source discovery such as RSS feeds, sitemaps, and other structured formats

New York Times Newsgathering bot collects public, non-copyright data for newsroom use.

NewsBlur RSS News fetcher operated by NewsBlur

Noibu JS Beautifier is a microservice operated by Noibu that downloads and formats JavaScript files for error solving.

Nooshub is an RSS Reader bot operated by Nooshub that fetches RSS/Atom feeds.

Omnisend bot checks integration in WooCommerce shops and Shopify, operated by Omnisend.

OpenRSS is an RSS feed provider and fetcher operated by OpenRSS, offering a healthier alternative to algorithmic feeds.

OurFamilyWizard bot syncs external iCalendars into Cozi calendars.

Overcast is a podcast player application operated by Overcast Radio, fetching RSS feeds and audio files from podcast hosting servers.

The PayPal webhooks is part of Paypal's Instant Payment Notification message service, automatically notifying merchants of events related to Paypal transactions.

PinterestBot is Pinterest's web crawler that indexes content and scrapes Pin details to drive traffic and ensure accuracy.

Fetches podcast feeds, for playback in the Pocket Casts apps

Polar is a webhook service operated by Polar that sends HTTP requests to inform web servers about billing events.

Extract Content to Show Print Friendly version. Publishers typically embed our button - https://www.printfriendly.com/button - so that their visitors can view a Print Friendly Page and/or create a PDF

Pro Sitemaps is an online sitemap generator service operated by PRO Sitemaps, used for creating and maintaining sitemaps for websites.

Protopage is an RSS reader bot operated by Protopage Ltd, indexing RSS news headlines.

PWABuilder is a free, open-source tool from Microsoft that helps developers build progressive web apps.

Rakuten Image extraction bot, operated by Rakuten, retrieves product images from partner websites.

Razorpay Webhook bot operated by Razorpay, sends webhooks to ecommerce sites.

Readable is a text analysis tool operated by Added Bytes Ltd, focused on clarity and plain language.

Retool platform user agent

RSS API is a bot operated by Tibush GmbH that periodically requests and parses RSS feeds for customers to monitor changes.

rss2tg_bot is a notification RSS/Atom bot for Telegram operated by Yellow Rubber Duck Consulting.

Scour RSS Bot is a personalized RSS feed reader operated by Scour, using AI/ML to surface content matching user interests.

SendGrid bot operated by SendGrid, sends email event data to customer APIs.

ServerHunterSpider is a bot operated by Server Hunter that indexes hosting plans' price, specifications, and stock.

Shopify-Captain-Hook is a system used by Shopify to deliver webhooks. It sends automated messages to a user's server when specific events occur within their Shopify store.

Website accessibility, SEO, and content quality scanner

Skroutz ImageBot to fetch the individual product images.

Skroutz uses SkroutzBot web crawler to download XML feeds.

Slackbot fetches URLs shared in Slack channels to generate link previews and unfurls for team communication.

Snipcart is an e-commerce solution for developers operated by Snipcart.

Spark Shipping is an eCommerce automation software for retailers running WooCommerce, automating product data integration, inventory synchronization, order sending, and tracking updates.

SparkpostBot is a bot operated by Bird for automating email transactions on web server events.

Statabot is a bot operated by StataCorp LLC that searches for stata.toc files and indexes their contents.

StatistikAustria bot operated by Statistik Bot to collect product prices for Austria's official consumer price index.

StatsDroneBot is a statistics scraping and aggregating tool for iGaming affiliates, operated by StatsDrone.

Stripe bot, operated by Stripe, pushes real-time event data to customers' webhook endpoints.

Stripebot is a web crawler operated by Stripe, collecting data from user websites for services and financial regulation compliance.

Svix Webhooks is a scalable webhook platform operated by Svix Inc., featuring automatic retries, signature verification, and deep observability.

Termly bot scanners for site compliance

Trustly is a payment bot operated by Trustly Group AB, sending notifications to merchants about payment statuses.

Turnitin.com offers various services to the educational community. Most prominently, we provide a widely used and effective plagiarism detection service. Part of the plagiarism prevention service relies on comparing student papers to content found on the Internet. Since we do not know ahead of ti...

Twilio Proxy bot operated by Twilio, Inc. for sending and receiving SMSes, making calls, and managing customer engagement.

upday is a news aggregator app bot operated by upday GmbH & Co. KG, crawling news sources to collect and index articles for user recommendations.

VaultPress is a WordPress backup service by Automattic, offering daily and real-time backups.

Crawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages.

Job wrapping data processor handling jobs distribution from employer websites to multiple endpoints, like job boards, advertisement platforms, job alerts etc.

WMF Citoid is a Wikimedia Foundation service that generates citations from URLs, DOIs, and ISBNs.

WMF Zotero Translation Server, operated by Wikimedia Foundation, extracts citation data using Zotero translators.

This is the major egress IP for our containerised WordPress platform so it is likely to be many flavours of WordPress and the potential to be any domain.

The Worldline Bot, operated by Worldline, handles notifications and callbacks related to payment processing.

Payment confirmation callbacks to ecommerce backends

We offer WOVN.io, a service for localization websites. We run a crawler to get source language of our clients' websites

Our plugin WPTimeCapsule is installed in more 30000 WordPress sites. When our backup servers send requests to trigger the backup on the WordPress sites, it is being blocked

WP Umbrella Bot is a tool for managing and maintaining WordPress websites, operated by WP Umbrella.

The XY Archive Compliance Bot is a website archiver operated by XY Archive Compliance for customers with archive compliance requirements.

Yokoy Group Webhooks is a bot operated by Yokoy Group AG, generating requests to book expenses or invoices to customer's ERP system.

Zapier is an automation tool operated by Zapier Inc. that moves information between web apps automatically.

Zendesk Webhook bot for development of Zendesk ticketing system and apps.