Home / Answers / Entity and data
Should you block AI crawlers?
If you are a local business trying to be recommended, no - blocking them removes you from the answers you are trying to appear in. The case for blocking applies to publishers whose content is the product. For a plumber or a dental practice, blocking is self-harm.
What blocking actually does
A disallow for GPTBot, ClaudeBot, PerplexityBot or Google-Extended tells those systems not to crawl your pages. They then describe you from other sources - directories, reviews, competitors' comparisons - or not at all.
You have not protected anything. You have handed the description of your business to third parties.
Who has a real reason to block
Publishers, course creators, anyone whose written work is the thing customers pay for. If a model can reproduce your product, crawling costs you sales. That is a genuine argument and not the situation of a service business.
Check what you are blocking already
Many sites block these crawlers by default - a theme, a security plugin, or a Cloudflare setting flipped on without anyone noticing. Read your own robots.txt before assuming you are open.
Also check your firewall. Cloudflare's AI bot controls operate above robots.txt and will block regardless of what the file says.
The middle ground
Allow crawling of your service and information pages; disallow anything genuinely private, like a client portal. That is normal practice, not a stance.
The short version
- Local businesses should allow AI crawlers
- Blocking hands your description to third parties
- Publishers have a real case; service businesses do not
- Check both robots.txt and your firewall settings
Related questions
Which crawlers should I name?
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Bingbot for Copilot.
Does allowing them mean my content is used for training?
In some cases yes. That is the trade for being described accurately.
Can I allow retrieval but not training?
Partly. OpenAI separates OAI-SearchBot from GPTBot, so you can allow one and block the other.
Keep reading
What is llms.txt and do you need one?
It is a plain-text file at the root of your site that summarizes what your business does and points to your...
Does schema markup help with AI search?
It helps, indirectly but genuinely. Schema states your facts in a form that cannot be misread - name, hours,...
Why isn't my business showing up in ChatGPT?
Almost always one of six things: the assistant has no location to work from, your Google profile is thin,...
Want this checked on your own business?
The free visibility check shows what AI assistants say about you today, where you sit on the map grid, and the three fixes worth doing first.
