Only 10 of 50 Major Travel Sites Set Explicit AI Crawler Rules

By Tom Whitfield

Only 10 of 50 major travel websites explicitly addressed AI crawlers in their robots.txt files, according to Skift’s review; the other 40 relied on catch-all bot rules or default settings.

For sellers, that gap creates uncertainty over whether AI systems can retrieve the prices, amenities and availability they need to represent a property or service in generated answers.

The proposal would turn that ambiguity into a concrete distribution choice for sellers. On Oct. 30, Anthropic, Google, Microsoft and OpenAI backed a proposal that would let a site opt out of AI training by all four with one explicit robots.txt entry. Catch-all rules would not carry that opt-out.

How much does AI traffic matter?

Adobe reported in July that AI visits to travel sites rose 119% year over year. The figure measures AI traffic, not bookings, revenue or an operator’s market share.

Adobe analyst Vivek Pandya called machine readability a major bottleneck. He wrote: “Significant portions of U.S. travel and retail sites remain unreadable to machines, restricting their presence across AI search results.”

That puts access directly into the distribution chain. If a seller’s content cannot be read, its offers may not appear where travelers ask AI to compare hotels, rates or availability. Pandya added that when an LLM cannot easily read brand content, including amenities, pricing or availability, “potential revenue is being left on the table.”

Skift’s audit did not calculate lost revenue, and Adobe’s traffic figure did not show how many AI referrals convert. The evidence establishes rising machine traffic and uneven access, not a measured booking lift.

Bottom line: AI referrals are growing faster than sellers can prove their commercial value.

Which sites opened the door?

Among the 10 sites with explicit policies, the general pattern favored visibility over training. Expedia and Vrbo gave AI search crawlers broad access, while Tripadvisor blocked training crawlers but allowed some search access. Best Western and ITC Hotels explicitly welcomed OpenAI’s training crawler.

The split matters because permission to train a model differs from permission to retrieve content for an answer. Broad search access can support distribution of a seller’s rates and details, while a training block can limit reuse of that content. The audit provides no ranking, click or conversion data.

Airbnb made the sample’s only major policy change. It allowed GPTBot and ClaudeBot in August, then disallowed access across the board by October. The reversal shows how quickly a seller’s route into AI channels can change.

Bottom line: Travel sellers are choosing selectively between model training and search discovery, rather than following one standard.

Do robots.txt rules match enforcement?

Eight sites, mainly hotels, returned 403 Forbidden errors to Skift’s automated requests. CraftedStays separately found that Marriott, Hilton, IHG, Hyatt and Four Seasons named no AI crawler on their banned lists, while their bot protection refused fetches outright.

That separates published policy from practical enforcement. A permissive robots.txt file does not guarantee retrieval of inventory, and a restrictive file may not identify every AI technology a site wants to exclude. For travel sellers, enforced access determines whether product content reaches AI search systems, not merely what the published file says.

Cloudflare data adds a measure of crawler demand. Meta’s crawler accounted for 25.5% of AI-bot requests to travel and tourism sites, ahead of OpenAI’s or Anthropic’s. That represents crawler-request share, not booking share, and it does not show which AI interface a traveler used.

Bottom line: The request can reach the site without the published policy matching what the bot actually encounters.

What will the joint opt-out change?

The four-provider proposal would standardize the training opt-out, but it would not settle search access. The 40 sites relying on catch-all or default rules would need an explicit entry to use the proposed protection, and the audit found that posted rules did not always match enforcement.

For travel sellers, that means three separate choices: machine access for discovery, retrieval for AI answers, and permission for model training. The joint proposal addresses only the third choice; it does not determine whether systems can retrieve a seller’s current rates or availability.

The next commercial test is whether clearer policies produce machine-readable listings, AI-search visibility, leads and completed bookings. The audit and Adobe’s 119% traffic figure do not yet measure those outcomes.

Bottom line: The next test for travel sellers is whether explicit crawler rules and enforced access convert AI visibility into leads and bookings.