All articles
Digital Marketing Strategy

The Robots.txt Line That's Silently Blocking Plumbers and Electricians From Every AI Search Answer

A single disallow line in your robots.txt file can keep ChatGPT, Perplexity, and Google's AI Overview from ever reading your plumbing or electrical site, no matter how good your content is.

3 min readUpdated 2026

A homeowner asks ChatGPT which electrician to call for a same day callout. Three names come back. A fourth company, with a faster response time and genuinely better reviews, never gets mentioned, not because the content on its site is weak, but because its own robots.txt file told the model's crawler to stay out before it ever reached a single page.

Your Robots.txt File Decides Who Gets To Read You

Every website has a robots.txt file sitting at its root address, written years ago to tell Google and Bing which pages to crawl. Most trades sites never touch it again. The problem is that newer crawlers such as GPTBot, ClaudeBot, PerplexityBot, and CCBot read that same file, and a line written for search engines in 2019 can silently shut all four of them out today.

1. The Four Crawlers Worth Checking For

  • GPTBot, which feeds ChatGPT's answers
  • ClaudeBot, which feeds Claude's answers
  • PerplexityBot, which feeds Perplexity's answers
  • CCBot, which feeds the open training sets several AI search tools draw on
38%of the trades websites ZENETERA audited in 2026 had blocked at least one of these four crawlers in robots.txt without the owner knowing the rule was there

2. Where The Block Usually Comes From

Nobody sits down and decides to block ChatGPT on purpose. The rule usually arrives through a site builder's discourage search engines checkbox left switched on from a staging version of the site, a security plugin that disallows every crawler except a short approved list, or a rebuild job where the old robots.txt file got copied across without anyone reading what was inside it.

What the file usually saysWhat it actually does
User-agent: * Disallow: /admin/Normal housekeeping, leaves your public pages open to every crawler
User-agent: GPTBot Disallow: /Added once by a plugin or a developer, blocks ChatGPT from ever reading the site

3. Checking Your Own Site Takes Two Minutes

Open your own site's address followed by /robots.txt in a browser tab. Read through it for a line naming GPTBot, ClaudeBot, PerplexityBot, or CCBot. If any of those names sit under a rule that reads Disallow: /, that crawler is turned away at the door, no matter how good the page it would have read actually is.

A robots.txt file that blocks AI crawlers does not make your business more private. It makes it invisible to the one channel increasingly deciding who gets called first.

The Takeaway

Fixing this is rarely more than deleting or narrowing a handful of lines. It will not rank a weak page higher and it will not fix bad content. What it does is stop a technical accident from quietly cancelling out every other improvement you make for AI search, which is the part most trades businesses never think to check.

4crawler names worth searching for in your own robots.txt file before you spend another pound on AI search visibility

01Contact

Let's work together

Which area are you interested in?