The Robots.txt Line That's Silently Blocking Villa Listings From Every AI Trip Planner
A single disallow line in your robots.txt file can stop ChatGPT, Perplexity, and Google's AI Overview from ever reading your villa listing, no matter how stunning the property actually is.
Your villa website can have perfect photos, an accurate nightly rate, and a fully built FAQ page, and still never appear in a single AI trip planner answer. The reason is often one line in a file almost no villa owner has ever opened: robots.txt. If that file blocks the wrong crawler, ChatGPT, Perplexity, and Google's AI Overview never see the page at all. They do not partially read it or rank it lower. They skip it completely.
1. What Robots.txt Actually Controls
Robots.txt sits at the root of your domain, at yourdomain.com slash robots.txt. It is a plain text file that tells automated crawlers which parts of your site they are allowed to fetch. Search engines have respected it for decades. AI companies built their own crawlers and gave most of them their own name in that same file, which means a rule written years ago for Google may say nothing about the bot that feeds ChatGPT today.
2. The Crawlers That Actually Feed AI Trip Planners
- GPTBot, which feeds ChatGPT and OpenAI's other products
- PerplexityBot, which Perplexity uses to answer travel questions directly
- Google Extended, which controls whether your pages can be used in Google's AI Overview and Gemini, separately from classic Googlebot
- ClaudeBot, used by Anthropic's Claude
- Bingbot, which still feeds Microsoft Copilot answers
Blocking classic Googlebot and blocking Google Extended are two different settings. A villa site can rank normally on Google search while still being invisible to the AI Overview that increasingly answers travel questions before a human ever clicks a result.
3. How a Villa Site Ends Up Blocking Itself
Three situations cause this far more often than a deliberate choice. A staging site that was never meant to be public gets a blanket disallow rule, and that same robots.txt file gets copied over when the real site launches. A website builder or booking plugin ships with a default robots.txt template written before AI crawlers existed, and it blocks everything not explicitly allowed. Or a developer added a rule to stop image scrapers and aggregator sites from copying villa photos, and that rule swept up every AI crawler along with them.
4. The Fix, Checked In Under A Minute
Open yourdomain.com slash robots.txt in any browser. If the file does not exist, nothing is blocked by default and you can move on. If it exists, look for any line that reads Disallow colon slash under a User agent block, then check which user agent that block applies to. A Disallow colon slash under User agent star blocks everyone, including every AI trip planner bot, unless a more specific rule further down the file explicitly allows one back in.
| User agent line | What it means for AI trip planners |
|---|---|
| User agent star, Disallow slash | Every crawler is blocked, including every AI trip planner bot |
| User agent GPTBot, Disallow slash | ChatGPT specifically cannot read the site, even if other crawlers can |
| User agent Google Extended, Disallow slash | Google's AI Overview and Gemini cannot use the site, while classic Google search still can |
| No robots.txt file at all | Nothing is blocked by default, every crawler is free to read the site |
5. What To Allow And What To Still Block
You do not need to open the entire site. Keep admin folders, booking system backends, and payment pages disallowed, the same way you always would. Make sure any blocking rule only targets those specific paths, never an entire user agent across the whole domain, and explicitly allow GPTBot, PerplexityBot, Google Extended, ClaudeBot, and Bingbot if a blanket rule exists at all. A short, specific allow list for the pages that describe the villa, the pricing, and the amenities is enough.