Before any AI visibility work, confirm the crawlers can actually read your site. This checks your robots.txt and tells you which AI crawlers are allowed and which are blocked. It takes a second and there is nothing to sign up for.
Why this matters
Watch: can AI see your business?
Fifty-six seconds on why ranking well is no longer the same thing as being found, and what to check first.
AI assistants build their answers from content they were able to read. If your robots.txt blocks GPTBot, then ChatGPT cannot use your site as a source, no matter how good it is or how well it ranks on Google. You simply do not exist in that conversation.
The blocks are usually accidental. A security plugin ships with an aggressive default, a hosting company adds a rule to reduce bandwidth, an agency copies a robots.txt from a template, or somebody blocked scrapers in 2023 when that felt prudent and nobody revisited it. The business then spends months wondering why assistants recommend competitors.
What each crawler does
- GPTBot collects content used to train OpenAI models. Blocking it keeps your content out of training data but does not stop ChatGPT citing you live.
- OAI-SearchBot powers ChatGPT search results. Blocking this one does remove you from ChatGPT's answers.
- ChatGPT-User fetches a page when a user asks ChatGPT to look at it directly.
- ClaudeBot is Anthropic's crawler for Claude.
- PerplexityBot indexes for Perplexity, which cites sources unusually prominently.
- Google-Extended controls whether your content can be used by Gemini and grounding in AI Overviews. It does not affect normal Google rankings.
- CCBot is Common Crawl, an open dataset that many AI systems draw on indirectly.
What it does not tell you
This reads robots.txt only. A crawler can also be blocked at the firewall, by Cloudflare bot rules, or by server configuration, none of which are visible from outside. If the checker says you are open but assistants still never mention you, the block may be one of those, or the honest answer may be that access was never your problem and there is simply nothing on the site worth citing.
Our GEO guide covers what to do once access is confirmed, and how to get cited by LLMs covers what makes a page worth quoting in the first place.
Common questions
Should I allow every AI crawler?
For most businesses that want to be found, yes. Publishers with content to protect reasonably take a different view, and the sensible middle is to allow the search-and-citation crawlers while blocking the training-only ones. The table above marks which is which.
How long until a change takes effect?
Crawlers re-read robots.txt within days, but appearing in answers takes longer, because the content still has to be crawled, indexed and judged worth citing. Weeks rather than hours, and unblocking alone changes nothing if there was never anything citable there.
What is llms.txt?
A proposed file, not yet a standard, that points AI systems at your most useful content in a readable form. Adoption is patchy and support is not guaranteed, so treat it as cheap insurance rather than a fix. We publish one; it took ten minutes.