AI Crawler Checker
Enter your URL. We read your robots.txt and llms.txt and show per AI crawler whether it has access.
Why AI crawlers matter
ChatGPT, Perplexity and Google AI Overviews can only cite you if their crawlers are allowed to read your pages. Many sites block them without knowing: an old security plugin, a CDN setting or a rule once added against 'scrapers'.
Training versus search
Not every crawler does the same job. GPTBot and Google-Extended collect data for training; OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages to answer questions live. Allowing only the search crawlers keeps you citable without contributing to training.
How to allow a crawler
Add a block to robots.txt with 'User-agent:' followed by the crawler's name and 'Allow: /'. Then check your firewall or CDN as well: since 2025 Cloudflare blocks AI bots by default on new domains.
What is llms.txt?
llms.txt is a simple text file in the root of your site that tells language models what you do and which pages matter most. It is not an official standard, but it takes minutes to create and helps AI systems understand your site faster.
Frequently asked questions
Want this done automatically every day?
The Autopilot handles the GEO layer for every new article it publishes.
Request early access