AI User Agents List (updated Dec 2025)

ChatGPT, Claude, Perplexity... Identify which AI bots are crawling your site.

🔍 Browse AI Bots

Explore our database of 58 AI bots. Filter by category, or behavior to find what you need.

Missing a bot? Contact us to suggest new bots for our database.

CATEGORIES Filter by Type

BEHAVIOR Filter by Bot Behavior

Showing 58 AI bots

AI2Bot

Allen Institute for AI AI Training

Allen Institute for AI bot for academic research and model training

Mozilla/5.0 (compatible; AI2Bot/1.0; +https://allenai.org/)
AI2Bot
#ai2 #academic #research #training
✅ Respectful
📚 Docs →

Amazonbot

Amazon AI Assistant

Amazon bot to improve Alexa and AWS AI services

Amazonbot/0.1 (+https://developer.amazon.com/support/amazonbot)
Amazonbot
#amazon #alexa #aws #assistant
✅ Respectful
📚 Docs →

Andibot

Andi AI Search

Andi AI search engine bot, competitor to Perplexity

Mozilla/5.0 (compatible; Andibot/1.0)
Andibot
#andi #search #answer-engine #competitor
✅ Respectful

anthropic-ai

Anthropic AI Training

Training bot for Anthropic's Claude models, collects data to improve models

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Web/1.0; +https://www.anthropic.com)
anthropic-ai
#claude #anthropic #training #bulk-data
✅ Respectful
📚 Docs →

Anthropic-Claude

Anthropic AI Assistant

Updated Anthropic Claude bot for real-time web access and citations

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Anthropic-Claude/1.0; +https://www.anthropic.com)
Anthropic-Claude
#anthropic #claude #realtime #citations
✅ Respectful
📚 Docs →

Claude-Web

Anthropic AI Search

Claude's web bot for exploration and indexing of web content

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Web/1.0; +https://www.anthropic.com)
claude-web
#claude #anthropic #web #crawling
✅ Respectful
📚 Docs →

ClaudeBot

Anthropic AI Assistant

Bot used by Claude to fetch citations and references in real-time during conversations

ClaudeBot/1.0; +https://www.anthropic.com
ClaudeBot
#claude #anthropic #citations #assistant
✅ Respectful
📚 Docs →

Applebot-Extended

Apple AI Training

Bot for training Apple AI models (Apple Intelligence)

Mozilla/5.0 (compatible; Applebot-Extended/1.0)
Applebot-Extended
#apple #apple-intelligence #training #siri
✅ Respectful
📚 Docs →

bigsur.ai

BigSur AI AI Training

New emerging AI bot, details on usage still limited

Mozilla/5.0 (compatible; bigsur.ai/1.0)
bigsur.ai
#bigsur #emerging #new #training
✅ Respectful

Brightbot

Bright Data AI Training

Bright Data analysis bot to collect data for AI

Mozilla/5.0 (compatible; Brightbot/1.0)
Brightbot
#bright-data #analysis #data-collection #training
✅ Respectful

Bytespider

ByteDance AI Training

ByteDance (TikTok) bot for training their Chinese AI models

Mozilla/5.0 (compatible; Bytespider; [email protected])
Bytespider
#bytedance #tiktok #chinese #training
✅ Respectful

TerraCotta

Ceramic AI Training

Ceramic AI crawler for web content indexing and model training, first seen June 2025

TerraCotta https://github.com/CeramicTeam/CeramicTerracotta
TerraCotta
#ceramic #training #emerging #crawler
❌ Ignores robots.txt
📚 Docs →

Character-AI

Character.AI AI Assistant

Character.AI bot for training conversational AI characters

Mozilla/5.0 (compatible; Character-AI/1.0; +https://character.ai/)
Character-AI
#character-ai #conversational #characters #training
✅ Respectful
📚 Docs →

Devin

Cognition AI AI Assistant

Devin AI code assistant bot to analyze and understand online code

Mozilla/5.0 (compatible; Devin/1.0)
Devin
#devin #code-assistant #programming #cognition-ai
✅ Respectful

Cohere-Ai

Cohere AI Training

Cohere bot for training their language models and NLP

Mozilla/5.0 (compatible; Cohere-AI/1.0; +https://cohere.com/)
Cohere-Ai
#cohere #nlp #training #enterprise
✅ Respectful
📚 Docs →

Cohere-Command

Cohere AI Assistant

Cohere Command model bot for real-time information retrieval

Mozilla/5.0 (compatible; Cohere-Command/1.0; +https://cohere.com/)
Cohere-Command
#cohere #command #assistant #enterprise
✅ Respectful
📚 Docs →

CCBot

Common Crawl AI Training

Common Crawl bot, widely used for training open source AI models

CCBot/2.0 (https://commoncrawl.org/faq/)
CCBot
#common-crawl #open-data #training #dataset
✅ Respectful
📚 Docs →

Crawlspace

Crawlspace AI Training

Crawling service specialized for AI and data extraction

Mozilla/5.0 (compatible; Crawlspace/1.0)
Crawlspace
#crawling-service #data-extraction #ai #training
✅ Respectful

DeepseekBot

DeepSeek AI Training

DeepSeek AI bot for training their advanced reasoning models and data collection

Mozilla/5.0 (compatible; DeepseekBot/1.0; +https://www.deepseek.com/bot)
DeepseekBot
#deepseek #reasoning #training #chinese
✅ Respectful
📚 Docs →

Diffbot

Diffbot AI Training

Diffbot bot for structured data extraction and creating knowledge graphs for AI

Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com/our-apis/crawler/)
Diffbot
#diffbot #knowledge-graph #extraction #structured-data
✅ Respectful
📚 Docs →

DuckAssistBot

DuckDuckGo AI Assistant

DuckDuckGo bot for their privacy-respecting AI assistant

Mozilla/5.0 (compatible; DuckAssistBot/1.0; +https://duckduckgo.com/duckassist)
DuckAssistBot
#duckduckgo #privacy #assistant #search
✅ Respectful
📚 Docs →

FirecrawlAgent

Firecrawl AI Training

New scraping service specialized for AI and LLMs

Mozilla/5.0 (compatible; FirecrawlAgent/1.0)
FirecrawlAgent
#firecrawl #scraping #llm #training
✅ Respectful

Bard-Ai

Google AI Assistant

Google Bard AI assistant bot for web content retrieval

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bard-AI/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
Bard-Ai
#google #bard #assistant #search
✅ Respectful
📚 Docs →

Gemini-Ai

Google AI Assistant

Google Gemini AI model bot for training and web content analysis

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-AI/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
Gemini-Ai
#google #gemini #training #analysis
✅ Respectful
📚 Docs →

Gemini-Deep-Research

Google AI Assistant

Bot for Gemini Deep Research in-depth searches

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research/1.0)
Gemini-Deep-Research
#google #gemini #deep-research #assistant
✅ Respectful

Google-CloudVertexBot

Google AI Training

Google crawler for Vertex AI Agents, crawls content at request of site owners building AI agents

Mozilla/5.0 (compatible; Google-CloudVertexBot/1.0; +https://cloud.google.com/vertex-ai)
Google-CloudVertexBot
#google #vertex-ai #training #agents
✅ Respectful
📚 Docs →

Google-Extended

Google AI Training

Token to control access to content for Gemini/Bard and Vertex AI

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
Google-Extended
#google #gemini #bard #vertex-ai
✅ Respectful
📚 Docs →

Google-NotebookLM

Google AI Assistant

Google NotebookLM bot that fetches individual URLs provided by users as sources for their research projects

Mozilla/5.0 (compatible; Google-NotebookLM/1.0; +https://notebooklm.google.com/)
Google-NotebookLM
#google #notebooklm #user-triggered #research #assistant
❌ Ignores robots.txt
📚 Docs →

GoogleAgent-Mariner

Google AI Assistant

Google Project Mariner agentic browser for AI Ultra subscribers ($249.99/month). Operates on cloud-based virtual machines as a remote browser environment rather than traditional crawler.

GoogleAgent-Mariner
GoogleAgent-Mariner
#google #mariner #agentic-browser #premium #cloud-vm
✅ Respectful
📚 Docs →

Groq-Bot

Groq AI Training

Groq inference engine bot for high-speed AI model data collection

Mozilla/5.0 (compatible; Groq-Bot/1.0; +https://groq.com/)
Groq-Bot
#groq #inference #high-speed #training
✅ Respectful
📚 Docs →

HuggingFace-Bot

Hugging Face AI Training

Hugging Face bot for training open-source AI models and datasets

Mozilla/5.0 (compatible; HuggingFace-Bot/1.0; +https://huggingface.co/)
HuggingFace-Bot
#huggingface #open-source #training #datasets
✅ Respectful
📚 Docs →

IbouBot

Ibou.io AI Search

Ethical search engine crawler that drives traffic to original sources. Uses GenAI for query processing but does NOT train AI models. Respects creators and publisher rights

Mozilla/5.0 (compatible; IbouBot/1.0; [email protected]; +https://ibou.io/iboubot.html)
IbouBot
217.113.196.0/24
#ibou #french #ethical-search #traffic-driver #creator-friendly
✅ Respectful
📚 Docs →

FacebookBot

Meta AI Training

Traditional Facebook bot extended for AI and machine learning

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
FacebookBot
#meta #facebook #social #ai
✅ Respectful
📚 Docs →

Meta-ExternalAgent

Meta AI Training

Meta bot for training their AI models (Llama, etc.)

Meta-ExternalAgent/1.0 (+https://developers.facebook.com/docs/sharing/bot)
Meta-ExternalAgent
#meta #facebook #llama #training
✅ Respectful
📚 Docs →

meta-webindexer

Meta AI Search

Meta web indexer bot for building independent search capabilities for Meta AI chatbot

meta-webindexer/1.1
meta-webindexer
#meta #search #indexing #ai-search
✅ Respectful
📚 Docs →

BingBot

Microsoft AI Search

Microsoft Bing crawler used for Bing Search and Copilot AI features

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 Edg/W.X.Y.Z
bingbot
#microsoft #bing #copilot #search
✅ Respectful
📚 Docs →

MistralAI-User

Mistral AI AI Assistant

Mistral AI bot to retrieve citations in Le Chat

MistralAI-User/1.0
MistralAI-User
#mistral #le-chat #french #citations
✅ Respectful

ChatGPT Atlas

OpenAI AI Assistant
⚠️ STEALTH

OpenAI's agentic browser with integrated AI. Uses standard Chrome user-agent, making it completely indistinguishable from regular browser traffic. Cannot be blocked via robots.txt. Features "agent mode" for autonomous task completion.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36
ChatGPT-Atlas
#openai #chatgpt #atlas #agentic-browser #stealth #undetectable
❌ Ignores robots.txt
📚 Docs →

ChatGPT-Browser

OpenAI AI Assistant

ChatGPT web browsing bot for real-time web access during conversations

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-Browser/1.0; +https://openai.com/bot)
ChatGPT-Browser
#openai #chatgpt #browsing #realtime
✅ Respectful
📚 Docs →

ChatGPT-User

OpenAI AI Assistant

Bot used for real-time searches when a user asks a question to ChatGPT

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)
ChatGPT-User
#chatgpt #realtime #search #user-triggered
✅ Respectful
📚 Docs →

ChatGPT-User v2.0

OpenAI AI Assistant

Updated version of ChatGPT-User bot for real-time searches (since February 2025)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/2.0; +https://openai.com/bot)
ChatGPT-User-v2
#chatgpt #realtime #search #user-triggered #v2
✅ Respectful
📚 Docs →

GPTBot

OpenAI AI Training

Bot used by OpenAI to collect training data for ChatGPT and future GPT models

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
GPTBot
#chatgpt #training #openai #gpt
✅ Respectful
📚 Docs →

OAI-SearchBot

OpenAI AI Search

Specific indexing bot for ChatGPT Search, competitor to Google Search

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
OAI-SearchBot
#openai #search #indexation #chatgpt-search
✅ Respectful
📚 Docs →

Perplexity Stealth

Perplexity AI AI Assistant
⚠️ STEALTH

Perplexity uses headless browsers with Chrome user agents to bypass blocking

Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
Perplexity-Stealth
#perplexity #stealth #headless #chrome
❌ Ignores robots.txt
⚠️ Stealth Analysis →

Perplexity-User

Perplexity AI AI Assistant

Bot triggered when a user clicks on a link in a Perplexity response

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/bot)
Perplexity-User
#perplexity #user-triggered #realtime
❌ Ignores robots.txt
⚠️ Stealth Analysis → 📚 Docs →

PerplexityBot

Perplexity AI AI Search

Perplexity indexing bot to feed their AI search engine

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)
PerplexityBot
#perplexity #search #answer-engine #indexation
❌ Ignores robots.txt
⚠️ Stealth Analysis → 📚 Docs →

Replicate-Bot

Replicate AI Training

Replicate platform bot for AI model training and data collection

Mozilla/5.0 (compatible; Replicate-Bot/1.0; +https://replicate.com/)
Replicate-Bot
#replicate #platform #training #models
✅ Respectful
📚 Docs →

RunPod-Bot

RunPod AI Training

RunPod cloud platform bot for GPU-based AI training data collection

Mozilla/5.0 (compatible; RunPod-Bot/1.0; +https://runpod.io/)
RunPod-Bot
#runpod #gpu #cloud #training
✅ Respectful
📚 Docs →

ImagesiftBot

The Hive AI Training

Bot for reverse image search and training image generation models

Mozilla/5.0 (compatible; ImagesiftBot/1.0)
ImagesiftBot
#image-search #reverse-search #image-generation #training
✅ Respectful

TimpiBot

Timpi AI Training

Timpi bot for training their Large Language Models

Mozilla/5.0 (compatible; TimpiBot/1.0)
TimpiBot
#timpi #llm #training #search
✅ Respectful

Together-Bot

Together AI AI Training

Together AI platform bot for decentralized AI model training

Mozilla/5.0 (compatible; Together-Bot/1.0; +https://together.ai/)
Together-Bot
#together-ai #decentralized #training #platform
✅ Respectful
📚 Docs →

Kangaroo Bot

Unknown (China) AI Training

Chinese AI bot, origin and exact usage unknown

Mozilla/5.0 (compatible; Kangaroo Bot/1.0)
Kangaroo Bot
#chinese #unknown #training #suspicious
❌ Ignores robots.txt

PanguBot

Unknown (China) AI Training

Another Chinese AI bot, possibly linked to Pangu models

Mozilla/5.0 (compatible; PanguBot/1.0)
PanguBot
#chinese #pangu #training #unknown
❌ Ignores robots.txt

Cotoyogi

Unknown (Japan) AI Training

Japanese AI bot, specific usage unknown

Mozilla/5.0 (compatible; Cotoyogi/1.0)
Cotoyogi
#japanese #unknown #training #asia
✅ Respectful

AkiraBot

Unknown (Malicious) AI Training
⚠️ STEALTH

Malicious spam bot using OpenAI LLMs to generate custom spam messages for contact forms. Uses generic Chrome user-agent strings and residential proxies. Primarily targets customer support chats via Selenium automation.

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
AkiraBot
#spam #malicious #llm-powered #selenium #contact-form-spam
❌ Ignores robots.txt
📚 Docs →

Webzio-Extended

Webz.io AI Training

Webz.io bot that collects data to sell to AI companies for training

Mozilla/5.0 (compatible; Webzio-Extended/1.0)
Webzio-Extended
#webzio #data-broker #training #commercial
✅ Respectful

xAI-Bot

xAI AI Training

Elon Musk's xAI bot for training Grok and other AI models

Mozilla/5.0 (compatible; xAI-Bot/1.0; +https://x.ai/)
xAI-Bot
#xai #grok #elon-musk #training
✅ Respectful
📚 Docs →

YouBot

You.com AI Search

You.com AI search engine bot for indexing and answering questions

Mozilla/5.0 (compatible; YouBot/1.0; +https://you.com/bot)
YouBot
#you-com #search #answer-engine #ai
✅ Respectful
📚 Docs →

Love this free tool?

Share it with your network and help others control AI bot access!

❓ Frequently Asked Questions

Everything you need to know about AI crawlers and robots.txt

What is an AI crawler ?

An LLM bot that requests your pages for model training or instant answers. You tell it what to do with "User-agent:" lines in your robots.txt file.

Is User-agent: * enough?

No. A wildcard line should be a catch‑all. You can still list AI crawlers you want to block. Some of them ignore the * directive, and only respect their specific user agent.

What do you mean by "top user agents"?

The bots in this guide account for 95% of AI crawler traffic. These are the most commonly seen AI bots in server logs.

Are bots required to follow directives in robots.txt files?

No. Anthropic was criticized in 2024 for ignoring robots.txt directives, and Perplexity has bypassed these rules.

How do I know which AI bots are visiting my website?

Check your server logs, and analyze the user agents. Most web analytics tools and server log analyzers can show you bot traffic patterns.

Should I block or allow AI crawlers?

It depends on your content strategy:

  • Allow if you want your content to appear in AI search results and get referral traffic
  • Block if you're concerned about content being used for training, with no compensation
  • Selective approach: Allow assistant bots (ChatGPT-User, ClaudeBot) but block training bots (GPTBot, CCBot)

What's the difference between training bots and assistant bots?

Training bots (like GPTBot, CCBot) crawl websites to collect data, to train AI models. Assistant bots (like ChatGPT-User, ClaudeBot) fetch content in real-time when users ask questions, potentially driving referral traffic to your site.

How often should I update my robots.txt for AI bots?

Review your server logs monthly, to verify if new AI bots are crawling your website. Also bookmark this page. We frequently update our list it, as new AI crawlers are discovered.

📊 Monitor your website's health

Now that your site is optimized for AI, keep track of performance, affiliate links, status codes and more!

Try PageRadar for free Discover all features