Search Engine Crawlers
These bots crawl and index web pages for search engines. They determine which pages appear in search results.
Search engine crawlers are the foundation of web search. They systematically visit websites, analyze content, and build the search index that powers organic results. Many now also feed AI features like AI Overviews and Copilot answers.
47 Search Engine Crawler in our directory
- Algolia by Algolia
Indexes content for Algolia search engine
- All Africa Crawler by AllAfrica Global Media
All Africa Crawler, operated by AllAfrica Global Media, aggregates news from Africa.
- Baidu ADS Server Proxy by Baidu
Baidu's DDoS scrubbing proxy.
- Baiduspider by Baidu
Baiduspider is the search engine crawler for the search engine Baidu.
- Bingbot by Microsoft
Bingbot is a crawler from Microsoft that powers Bing search.
- Cốc Cốc by Coccoc
Cốc Cốc bot, operated by Coccoc, is a search engine bot that scrapes websites requested by the Vietnamese search engine Coc Coc.
- Coveo Bot by Coveo
Coveo Bot is a search engine bot operated by Coveo, providing relevant experiences to end users by crawling websites.
- Crawlson by Crawlson
Crawlson is a search engine crawler for the crawlson.com search engine.
- Dataprovider.com by Dataprovider.com
Dataprovider.com indexes the web and structures the data.
- Daum by Daum
Daum is a Korean search engine crawler operated by Daum.
- DuckDuckBot by DuckDuckGo
DuckDuckBot is a web crawler for DuckDuckGo. DuckDuckBot's job is to constantly crawl the web and find web pages to add to the DuckDuckGo search index.
- Freespoke by Freespoke
Freespoke is a search engine that believes in free speech and shows you all viewpoints.
- Funnelback by Squiz - FunnelBack
Funnelback is an enterprise search platform operated by Squiz, used for indexing content from an organization's websites and data repositories to power internal search functions.
- Gigabot by Gigablast
Gigablast is the only non-Big Tech search engine in the U.S. that uses its own search index and algorithms.
- Google Images by Google
Google Images bot is a search engine crawler operated by Google for Google Images Search.
- Google Scholar by Google
Google Scholar bot crawls and indexes scholarly literature from academic publishers, repositories, and university websites.
- Google Videos by Google
The Google Videos bot is a search engine crawler operated by Google for Google Video Search.
- Googlebot by Google
Google uses crawlers and fetchers to perform actions for its products, either automatically or triggered by user request.
- Hype Machine by Hype Machine
Hype Machine is a search engine bot operated by Hype Machine that monitors music publications/blogs for new artists.
- IbouBot by Babbar
IbouBot is a search engine crawler operated by Babbar, used to fuel and update the graph representation of the World Wide Web.
- jobswithgptcom-bot by jobswithgptcom
jobswithgptcom-bot is a crawler operated by jobswithgptcom, focusing on job postings for jobswithgpt.com.
- Kagi Bot by Kagi
Kagi Bot is a web crawler for the Kagi search engine, operated by Kagi, used to build its own search index for its ad-free search product.
- Level9SearchBot by Level 9
Level9SearchBot is a search engine bot operated by Level 9.
- Linespider by LINE Corporation
Linespider is a web crawler operated by LINE Corporation, providing search results for LINE services.
- MagiBot by Peak Labs
MagiBot is owned by Peak Labs which focuses on the research and development of information extraction and retrieval technology to transform knowledge in natural language into immeasurable value.
- Marginalia Search by Marginalia
Marginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs.
- Mars Finder by Mars Flag
Mars Finder is a website search service operated by Mars Flag, designed to utilize the maximum potential of a website.
- Mojeek by Mojeek
MojeekBot is a web crawler for the Mojeek search engine, operated by Mojeek.
- MRGbot by MRG Web Services srl
MRGbot is a search engine bot operated by MRG Web Services srl, aimed at generating a corpus of data.
- MSN by Microsoft
MSNBot is a web crawler operated by Microsoft for indexing web pages for MSN Search.
- Neevabot by Neeva
Neevabot is the web crawler for the search engine neeva.com.
- NewsNow by NewsNowUK
NewsNow bot is a web crawler for the news aggregator service NewsNow, operated by NewsNowUK.
- Notabot by Helpfeel
Notabot is a crawler operated by Helpfeel for integrating external search engine.
- Potions by Potions
The Potions bot, operated by Potions, fetches product feeds and crawls e-commerce data.
- Qwantbot by Qwant
Qwantbot is a web crawler operated by Qwant, a European search engine that prioritizes user privacy, used for search indexing and web content discovery.
- Seekport by SISTRIX
Seekport is a search engine bot operated by SISTRIX, providing a public, free, and independent alternative to Google.
- Seznam by Seznam
SeznamBot is a search engine crawler operated by Seznam for search indexing.
- SiteSearch360 by SEMKNOX GmbH
SiteSearch360 is a search engine bot operated by SEMKNOX GmbH, used for indexing content on customer sites for search.
- SMTnet PM Bot by SMTnet
SMTnet PM Bot is a search engine crawler operated by SMTnet, crawling partner company websites for on-site search engine inclusion.
- Sogou by Sogou
Sogou spider is a search engine crawler operated by Sogou, used for indexing web content.
- Toutiao by Bytedance
Toutiao is ByteDance's automated news aggregation bot collecting content across web platforms.
- Yahoo Slurp by Yahoo
Yahoo Slurp is a search engine crawler operated by Yahoo, used for indexing web pages and collecting content.
- Yahoo! JAPAN by Yahoo! JAPAN
Yahoo! JAPAN manages and operates a system that accesses web pages published on the Internet for the purpose of providing services, research, development, maintenance, etc.
- YandexBot by Yandex
The main indexing robot for Yandex search.
- Yeti by Naver
Yeti is a web crawler operated by Naver, a South Korean search engine, used for indexing websites to provide search results.
- ZoomInfo by ZoomInfo
Zoominfobot is an indexing robot for a web search engine, similar to Google. Created by Zoom Information Inc.(www.zoominfo.com), Zoominfobot’s patented technology continually scans millions of corporate websites, press releases, electronic news services, SEC filings and other online sources. Usin...
- ZumBot by Zum Internet Corp
ZumBot is a web crawler operated by Zum Internet Corp for indexing webpages for Zum Open Internet Search.