Last month I analyzed the server logs of an e-commerce client and saw something striking: GPTBot was making 40,000 requests per month, ClaudeBot was at 12,000, and PerplexityBot stopped by every few minutes. None of those crawlers were listed in their robots.txt. They did not even know these bots existed, let alone what to do about them. That is the problem most companies are running into now: your robots.txt was probably written for Googlebot and Bingbot, while a whole new wave of crawlers is indexing your site for ChatGPT, Claude and Perplexity.
This article is a practical guide, not theory. We will show you exactly which user-agents you need to know, how to configure them, and share our own implementation (app/robots.ts) that we use on WeArkMedia projects.
Why robots.txt works differently now than it did five years ago
Robots.txt has existed since 1994 and has always done one thing: tell crawlers which paths they may or may not visit. Simple, static, predictable. The problem is that the assumptions behind that file no longer hold.
Traditional crawlers like Googlebot index your pages to rank them in a list of results. In return, you get click traffic. AI crawlers work fundamentally differently: GPTBot, ClaudeBot and PerplexityBot retrieve content to process it into training data or to generate real-time answers inside a chat interface. The user may see your brand name, but often never clicks through to your site.
That changes the trade-off completely. If you block these bots, your business simply does not exist in the answer a user gets when they ask ChatGPT or Perplexity about products in your sector. If you allow them all without thinking, you may be giving away your entire content library with no control over how it is used. The right strategy lies somewhere in the middle, and you need to choose it deliberately, not let it happen by accident because your robots.txt was never updated.
To understand how these crawlers find and process your site, this GEO guide on how ChatGPT finds your website is a good companion to this article.
The key AI crawlers you need to know
Before you configure anything, you need to know who is actually stopping by. Here are the crawlers that matter most in 2026.
GPTBot (OpenAI)
GPTBot is the crawler OpenAI uses to collect content for training and to generate answers in ChatGPT, including browsing functionality. User-agent string: GPTBot. OpenAI publishes this crawler's IP ranges, which makes it easier to verify traffic in your logs.
ClaudeBot (Anthropic)
ClaudeBot crawls for Anthropic's Claude models. There is also a variant, anthropic-ai, used specifically to retrieve content during user interactions (such as Claude's web search). Both user-agents are worth managing separately because they serve different purposes.
PerplexityBot
Perplexity has become a serious source of AI traffic, especially because it gives answers with direct source citations, often including a clickable link to your site. That makes PerplexityBot one of the few AI crawlers where allowing access almost always pays off, because it actually produces referral traffic rather than only content extraction.
Other relevant crawlers
- Google-Extended — controls whether Google may use your content for Gemini and AI Overviews, separately from regular Googlebot indexing.
- CCBot — the Common Crawl crawler, whose dataset is used by countless AI labs to train models.
- Applebot-Extended — governs use of your content for Apple Intelligence.
- Bytespider — tied to ByteDance (TikTok), aggressive in crawl behavior and often a concern for server load.
Each crawler has a different purpose and a different business model behind it. That means a one-size-fits-all "allow all" or "disallow all" rule is almost always wrong.
Practical implementation: our own robots.ts
On WeArkMedia projects using Next.js, we use a dynamically generated robots.txt file via app/robots.ts instead of a static file in the public folder. That gives us the flexibility to define and adjust rules per crawler without forcing a deploy for every change.
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
disallow: ['/admin/', '/api/', '/_next/'],
},
{
userAgent: 'GPTBot',
allow: '/',
disallow: ['/admin/', '/api/'],
},
{
userAgent: 'ClaudeBot',
allow: '/',
disallow: ['/admin/', '/api/'],
},
{
userAgent: 'PerplexityBot',
allow: '/',
},
{
userAgent: 'Google-Extended',
allow: '/',
},
{
userAgent: 'Bytespider',
disallow: '/',
},
{
userAgent: 'CCBot',
disallow: '/',
},
],
sitemap: 'https://wearkmedia.com/sitemap.xml',
}
}
Why these choices
We deliberately allow GPTBot, ClaudeBot and PerplexityBot on all public content, because our clients want to be visible in AI answers, which is the core of GEO/AEO work. We block Bytespider by default because this crawler is known for aggressive crawl behavior without clear value for most Western businesses. We often block CCBot as well, unless a client specifically wants to contribute to open training datasets, because it is unclear which model ultimately benefits from that data, and no direct attribution comes back.
Important detail: robots.txt is a request, not a lock on the door. Serious crawlers such as GPTBot and ClaudeBot generally respect the rules, but there are also less scrupulous scrapers that spoof the user-agent or ignore the file. For real protection of sensitive content, you need server-side rate limiting or a WAF. Robots.txt only governs the behavior of bots that identify themselves properly.
Strategy: when blocking actually makes sense
Not every situation calls for full openness. There are three scenarios where blocking AI crawlers is the better choice.
Content behind a paywall or login
If you have premium content, course materials or customer portals, you do not want GPTBot taking it without anyone ever paying for it. Block these paths specifically in the per-crawler disallow rules rather than closing off your entire domain.
Sites that rely solely on advertising revenue
For publishers and content sites that live on page views and ad impressions, content leaking into AI answers can directly cost revenue. This requires a deliberate trade-off between visibility in AI answers and direct revenue from visitors.
Server load on smaller hosting plans
Some AI crawlers, especially the lesser-known ones, crawl inefficiently and can put heavy strain on shared hosting environments. If you notice server costs rising from bot traffic without clear benefits in return, selective blocking is a legitimate choice.
For most SMBs and service providers, however, the opposite is true: allowing access is the norm, because visibility in AI answers is exactly the goal of modern AEO strategies. A locked-down robots.txt is then self-sabotage for your visibility, no matter how good your content is.