article · IS MY WEBSITE BLOCKING AI CRAWLERS
Check It Yourself InTen Minutes.
A good share of the sites I audit in The Valley block at least one of the crawlers that feed AI answers, and nobody in the building knows it happened. This is the check, in the order I run it, with the file paths so you can do it without me.
trusted by: UFC, Caesars Entertainment, Churchill Downs, Neighborly, Supercuts, City of Las Vegas, SDMI, Switch, Silverton, Panda Express, TAO, Ellie Mental Health, Dollar Loan Center, Dot Vegas Domains, HFC, Lithion Battery
Trusted by UFC, Caesars Entertainment, City of Las Vegas, Supercuts
“Justin's expertise, responsiveness, and genuine investment in our success have been evident throughout this process. We are truly grateful.”
01 · what are ai crawlers
What AI crawlers are.
Separate from Google's normal crawler, there is now a second set of automated readers, one per AI product. They are the reason your business shows up when somebody asks ChatGPT or Perplexity or Google's AI answer who to call. If one of them cannot fetch your pages, you are not in the answer.
These readers do two different jobs, and the distinction is the whole decision later on. Some fetch pages to train a model. Some fetch pages to answer the question a person just typed, and cite the source. Blocking the first costs you nothing today. Blocking the second removes you from the answer.
02 · how to check if chatgpt can read my website
Where to check for a block.
Five places, in this order. The first two catch most of it and take about a minute each.
Your robots.txt file
Type your domain followed by /robots.txt into a browser. Read for any Disallow line that sits under a User-agent line naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Applebot-Extended or Google-Extended. A Disallow of a single forward slash means the whole site.
Your firewall or security plugin
Robots.txt is a request. A firewall is a wall. Look in Cloudflare's bot settings, or your security plugin's bot list, for anything switched on that reads as blocking AI bots or unverified bots. This is the most common cause I find, and it never shows up in robots.txt.
The page source, for meta tags
A noai, noimageai or a robots noindex tag in the head of a template quietly covers every page built from that template. Check the templates that matter, usually the home page and the service pages.
Whether the content needs JavaScript to appear
Turn JavaScript off in your browser and reload. If your headline and your body copy vanish, several of these readers see the same empty page. Nothing is blocked and you are still absent.
Your server logs
The one check that gives you a fact instead of an inference. Filter the last thirty days for those agent names. If a name never appears, either it never came or it never got through, and both are worth knowing.
03 · why would a website block ai
How the block usually happens.
Almost never on purpose. In the audits I publish, the block traces back to one of four things, none of them a decision anybody in the business made.
A security plugin shipped an update with a broader bot list than the version before it. A host turned on bot protection at the account level. A developer copied a robots.txt from a staging site where blocking everything was correct. Or somebody read an article about protecting content from AI and switched off the part that gets you recommended at the same time.
04 · should i block ai crawlers
Whether to block AI crawlers.
There is a defensible reason to opt out of model training. A firm with proprietary written work, or a photographer whose images are the product, has something real to protect, and the training opt-outs exist for exactly that.
There is almost never a reason to block the retrieval readers. Those fetch a page because a buyer asked a question this minute, and they cite what they fetch. Blocking them is the same decision as asking Google to remove you from search, made without anyone noticing they made it.
So the answer for most professional practices is split, not all or nothing. Opt out of training if the content is the asset. Stay open to retrieval always.
| agent name | Who it belongs to | What blocking it costs you |
|---|---|---|
| GPTBot | OpenAI, crawling at scale | A training opt-out. Little to no visible cost today |
| OAI-SearchBot | OpenAI, the ChatGPT search index | You stop being a citable source inside ChatGPT |
| ChatGPT-User | OpenAI, fetching a page a user asked about | The one person actively looking at you gets nothing |
| ClaudeBot and Claude-User | Anthropic | You stop being citable in Claude |
| PerplexityBot | Perplexity | You leave the answer engine most likely to send a click |
| Google-Extended | Google, a training permission token | A training opt-out only. It does not affect Google search |
| Applebot-Extended | Apple, a training permission token | A training opt-out only. Apple search is unaffected |
05 · how to get cited by ai
What makes a page quotable.
Access is the floor, not the finish. Once the readers can reach the page, the pages that get quoted have a shape in common, and you can check it by eye.
06 · ai crawler questions
AI crawler questions.
Does blocking GPTBot remove me from ChatGPT?
Will allowing AI crawlers hurt my Google rankings?
How do I know it is fixed?
Is llms.txt worth adding?
Can you check this for me?
07 · ai search visibility las vegas
Where AI search visibility sits.
Being fetchable is one phase of the search work, not a project of its own. It lives beside the technical work, the pages themselves and the Google Business Profile, and I do all of it as one line.
AI search visibility
The phase that checks what the answer engines can read and cite.
Technical SEO
Crawlability, indexing and speed, checked line by line.
The written audit
Everything above, published at a URL you can open yourself.