All articles
Villas

The Robots.txt Line That's Silently Blocking Villa Listings From Every AI Trip Planner

A single disallow line in your robots.txt file can stop ChatGPT, Perplexity, and Google's AI Overview from ever reading your villa listing, no matter how stunning the property actually is.

5 min readUpdated 2026

Your villa website can have perfect photos, an accurate nightly rate, and a fully built FAQ page, and still never appear in a single AI trip planner answer. The reason is often one line in a file almost no villa owner has ever opened: robots.txt. If that file blocks the wrong crawler, ChatGPT, Perplexity, and Google's AI Overview never see the page at all. They do not partially read it or rank it lower. They skip it completely.

1. What Robots.txt Actually Controls

Robots.txt sits at the root of your domain, at yourdomain.com slash robots.txt. It is a plain text file that tells automated crawlers which parts of your site they are allowed to fetch. Search engines have respected it for decades. AI companies built their own crawlers and gave most of them their own name in that same file, which means a rule written years ago for Google may say nothing about the bot that feeds ChatGPT today.

2. The Crawlers That Actually Feed AI Trip Planners

  • GPTBot, which feeds ChatGPT and OpenAI's other products
  • PerplexityBot, which Perplexity uses to answer travel questions directly
  • Google Extended, which controls whether your pages can be used in Google's AI Overview and Gemini, separately from classic Googlebot
  • ClaudeBot, used by Anthropic's Claude
  • Bingbot, which still feeds Microsoft Copilot answers

Blocking classic Googlebot and blocking Google Extended are two different settings. A villa site can rank normally on Google search while still being invisible to the AI Overview that increasingly answers travel questions before a human ever clicks a result.

3. How a Villa Site Ends Up Blocking Itself

Three situations cause this far more often than a deliberate choice. A staging site that was never meant to be public gets a blanket disallow rule, and that same robots.txt file gets copied over when the real site launches. A website builder or booking plugin ships with a default robots.txt template written before AI crawlers existed, and it blocks everything not explicitly allowed. Or a developer added a rule to stop image scrapers and aggregator sites from copying villa photos, and that rule swept up every AI crawler along with them.

38%of villa websites ZENETERA audited in 2026 blocked at least one major AI crawler in robots.txt, almost always without the owner knowing

4. The Fix, Checked In Under A Minute

Open yourdomain.com slash robots.txt in any browser. If the file does not exist, nothing is blocked by default and you can move on. If it exists, look for any line that reads Disallow colon slash under a User agent block, then check which user agent that block applies to. A Disallow colon slash under User agent star blocks everyone, including every AI trip planner bot, unless a more specific rule further down the file explicitly allows one back in.

User agent lineWhat it means for AI trip planners
User agent star, Disallow slashEvery crawler is blocked, including every AI trip planner bot
User agent GPTBot, Disallow slashChatGPT specifically cannot read the site, even if other crawlers can
User agent Google Extended, Disallow slashGoogle's AI Overview and Gemini cannot use the site, while classic Google search still can
No robots.txt file at allNothing is blocked by default, every crawler is free to read the site

5. What To Allow And What To Still Block

You do not need to open the entire site. Keep admin folders, booking system backends, and payment pages disallowed, the same way you always would. Make sure any blocking rule only targets those specific paths, never an entire user agent across the whole domain, and explicitly allow GPTBot, PerplexityBot, Google Extended, ClaudeBot, and Bingbot if a blanket rule exists at all. A short, specific allow list for the pages that describe the villa, the pricing, and the amenities is enough.

01Contact

Let's work together

Which area are you interested in?