Search crawlers
Build the index an assistant searches when it answers. Block these and assistants can't find or cite your pages.
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
- Bingbot
Free tool
Can AI crawlers read your site?
See what your robots.txt allows GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers, and whether a live request as each of them gets your homepage or hits a block.
Why it matters
AI can only cite pages it can reach.
Assistants answer from pages their crawlers were allowed to read. Three kinds of crawler do different jobs, so the right rules differ.
User-request fetchers
Open a page live because a person asked about it, for example when someone pastes your link into ChatGPT.
- ChatGPT-User
- Claude-User
- Perplexity-User
Training crawlers
Collect content to train future models. Blocking them is a choice and doesn't remove you from today's answers.
- GPTBot
- ClaudeBot
- Google-Extended
- CCBot
robots.txt isn’t the whole story
robots.txt only asks crawlers to stay away. CDN bot settings, firewalls and server rules can refuse a crawler that robots.txt allows, often without anyone noticing. That’s why the checker also requests your homepage as each crawler.
What the check shows
Honest about what one request can tell.
It shows
- What your robots.txt declares for each crawler, with the exact group and lines that apply.
- Whether your server, CDN or firewall answers each crawler's user agent differently from a normal visitor: blocks (403, 429), challenges and timeouts.
- Rule-only tokens such as Google-Extended, which control AI use but never fetch pages.
It can’t show
- Whether the real crawlers visit. Our requests come from our server, not the vendors' IP addresses, and sites that verify IPs may treat the real crawler differently.
- Pages other than your homepage. robots.txt rules for other paths are listed, but only the homepage is requested.
- Whether assistants will cite you. Access is the first step, not a ranking.
See which AI crawlers really visit, verified by IP.
Connect your site through Cloudflare and Kliqo’s Crawl insights records every AI crawler visit, checks it against the vendor’s published IP ranges, and shows the pages they read and the errors they hit. 7-day free trial.
FAQ
Crawler questions.
What does the AI crawler checker check?
Two things for each AI crawler: what your robots.txt allows it to crawl, and what happens when we request your homepage once with its user agent. Together they show both your declared rules and blocks at your server, CDN or firewall that robots.txt doesn't show.
Why does robots.txt allow a crawler that the live request shows as blocked?
robots.txt is a request to well-behaved crawlers; it doesn't enforce anything. Firewalls, CDN bot settings (such as Cloudflare's AI bot blocking) and server rules can still refuse a crawler that robots.txt allows. The live request catches those cases.
Does a green result mean ChatGPT will cite my site?
No. It means the crawler isn't stopped by your rules and our request with its user agent got your homepage. Whether an assistant cites you depends on your content and many other sources. Our request comes from our server, not from the vendor's own IP addresses, so a site that verifies crawler IPs can treat the real crawler differently.
Should I block AI training crawlers like GPTBot?
It's a choice. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content for future models; blocking them doesn't remove you from today's answers. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether assistants can find and cite your pages now, so most sites keep those open.
Why do Google-Extended and Applebot-Extended say "Not a separate crawler"?
They are robots.txt tokens, not crawlers. Google and Apple fetch pages with Googlebot and Applebot and read these tokens to decide whether your content may be used for their AI models. So only the robots.txt verdict applies to them.
How often can I run it, and what do you store?
You can check 5 different sites a day. A site's result is saved for 24 hours and can be checked fresh once an hour. We store the domain, the result and a salted hash of your IP address for 30 days, nothing else. To see real crawler visits every day, Kliqo's Crawl insights reads them from your site, verified by IP.