AI & security
AI Crawler Checker
Which AI crawlers your robots.txt actually lets in
Every AI crawler is controlled by its own robots.txt token, and blocking one does nothing to the others. OpenAI alone operates three - GPTBot for training, OAI-SearchBot for search retrieval, ChatGPT-User for live browsing on a user's behalf - and blocking GPTBot while leaving the other two open is extremely common and almost always unintentional.
Try:
- Free, no signupNo account, no card
- No AI in the score986 deterministic rules
- Nothing publishedYour scans stay yours
- Answers in secondsQuick scan, no browser
So each token is checked individually rather than collapsed into one 'AI crawlers' verdict: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot and dozens more, alongside llms.txt.
The result tells you what you have actually decided, which is frequently not what you meant to decide. Blocking AI crawlers is a legitimate choice; blocking the training crawler while leaving the citation crawler open, or the reverse, usually is not.
Why this matters
You do not know what your robots.txt lets in any more
Most robots.txt files are a stack of pasted snippets. One from an article about blocking AI training, one from a plugin, one from whoever built the site three years ago. Meanwhile the token list keeps moving: OpenAI alone runs separate agents for training, for search and for live browsing, and blocking one leaves the other two open. The file takes ten seconds to read and still nobody can say what it currently permits.
What it costs you when this is wrong
You disappear from answers, not from Google
Disallow OAI-SearchBot or Claude-SearchBot and your pages stop being retrievable as answer sources, while Googlebot carries on and your traffic graph looks normal. You find out when someone else is cited in an answer about your own product.
The opt-out you think you made is partial
Blocking GPTBot does nothing to ClaudeBot, CCBot, Bytespider or meta-externalagent. If you blocked one token and stopped, the others are still crawling, and nothing tells you that the decision you made was a fraction of the one you intended.
A customer's assistant is turned away
Tokens such as ChatGPT-Agent, Operator and NovaAct are the ones acting for a real person asking about you right now. If they are disallowed, the assistant reports it cannot reach your site, and that lost visit never appears in your analytics at all.
Why run it here
81 crawlers, 81 separate verdicts
Every token is its own rule carrying its own weight: GPTBot and ClaudeBot are critical, ChatGPT-Agent and Claude-SearchBot are errors, CCBot and Bytespider are warnings. A single collapsed "AI crawlers" result cannot tell you which of those you have shut out.
Read the way a crawler reads it
Groups, specificity and Allow overrides are resolved by a conformance parser, so a wildcard Disallow followed by an Allow for GPTBot reports as allowed - the conclusion a crawler actually reaches. A fixture pins that exact case so it cannot regress; access is evaluated at the site root.
Allowed on purpose, or allowed by silence
Each verdict records whether robots.txt says anything at all about that token, or whether it is open only because nothing mentions it. That is the difference between a position you chose and one you inherited.
This check runs on its own here, and as part of the full audit alongside the other 985 rules. Either way the findings carry their evidence and their citation, and the report is never published or kept.
How to allow AI crawlers
Each operator obeys its own token, and robots.txt groups are not additive — a crawler follows the single most specific group matching its name and ignores every other.
List which AI crawlers you want to permit, deciding separately about training, search indexing and live retrieval. They are three different trades.
Add or remove the specific tokens: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-User and Claude-SearchBot for Claude; Google-Extended, PerplexityBot, Applebot-Extended and CCBot for the rest.
Repeat your site-wide Disallow rules inside any group you create. A group containing only Allow releases that crawler from the wildcard rules entirely, which is almost never intended.
Remember that Google-Extended governs Gemini and AI Overviews only and has no effect whatsoever on ordinary Google ranking.
Re-check after every robots.txt edit. This is the file where a one-line change goes unnoticed for months.
What happens when you run it
Enter your address
Just the domain is enough — example.com, with or without the https. No account, no card, no crawl of your whole site.
We render it like a visitor
Your page opens in a real browser and JavaScript runs to completion, so what gets audited is the page a person actually sees — not the raw HTML your server sent. A screenshot comes back with the report as evidence of exactly what we measured.
You get findings you can act on
Up to 986 checks run against that rendered page. Each finding names the rule, quotes the evidence found on your page, cites the public specification behind it, and says what to change. Nothing is a judgement call — run it twice on an unchanged page and the score is identical.
What runs this check
- RFC 9309 robots parsing
- Real browser rendering
Named because provenance is the product. Every finding in your report says which of these produced it, so you can check the reasoning rather than take a score on faith.
Questions
- Should I block AI crawlers?
- That is a business decision, not a technical one, and this tool does not take a side. It tells you which are currently allowed so the decision you have made is the one you intended. Blocking training while allowing retrieval is a coherent position; doing it by accident is not.
- Does blocking GPTBot remove me from ChatGPT?
- Not by itself. GPTBot is the training crawler. Search retrieval uses OAI-SearchBot and live browsing uses ChatGPT-User - different tokens, checked separately here.
- Is llms.txt a standard?
- No. It is a proposal, with no established support from any major model provider. It is checked and reported at EMERGING confidence, which means it appears in your report and is deliberately near-incapable of moving your score.
Also checks
This page answers these searches too — they are the same job, so they share one page rather than being split across near-identical ones:
- gptbot robots.txt
- block ai crawlers
- claudebot checker
- llms.txt checker
Other checkers
- HTML ValidatorW3C markup conformance, checked by a real parser
- CSS ValidatorStylesheets parsed and checked against the W3C property grammars
- Schema Markup ValidatorJSON-LD parsed, expanded to real schema.org IRIs, and checked
- Rich Results TestWhich search features your structured data qualifies for
- Website Scam DetectorThe trust signals people and payment processors check before believing a site
- AI ValidatorWhether AI models can find, read, quote and operate your site
- Broken Link CheckerEvery link on the page requested, with blocked and broken told apart
- SEO CheckerTitles, meta, canonicals, indexability and internal linking
- Accessibility CheckerTwo WCAG engines — one in a real browser, one on your markup
- Security Headers CheckerResponse headers, CSP quality, and the TLS certificate underneath
- Core Web Vitals CheckerMeasured for real, plus the page-weight causes behind the numbers
- AI Agent Readiness CheckerCan an agent find your buttons, read your labels and finish the task
- W3C ValidatorMarkup and stylesheet conformance in one pass
- Answer Engine CheckerWhether AI answer engines can quote you, and attribute it
- SSL Certificate CheckerA real handshake: certificate, expiry, protocol and cipher
- robots.txt CheckerWhat your robots.txt actually allows, group by group
- Meta Tag CheckerTitles, descriptions, canonicals and robots directives
- Open Graph CheckerHow your link looks when someone shares it
- Image SEO CheckerAlt text, dimensions, formats and lazy loading
- Mobile-Friendly TestViewport, tap targets and responsive behaviour
- Website CheckerEvery module, one scan, one prioritised fix list
Run the ai crawler checker now
No signup, no credit card. Quick answers in seconds; a full audit takes about a minute.
Run a free scanNo signup. No credit card. Nothing stored but the result.