AI Readiness of Fashion and Beauty Stores in 2026: We Checked 322 Brand Websites

As AI assistants become a way shoppers discover products, a practical question follows for every brand: when ChatGPT, Gemini, Claude or Perplexity look at our site, what can they actually read? To answer it at scale, we ran the same automated checks an AI crawler faces against 322 fashion and beauty brand websites in the United States, Europe and Australia in October 2026.

The headline is reassuring and uncomfortable at the same time. Reassuring, because fashion and beauty brands rarely shut AI crawlers out in robots.txt. Uncomfortable, because one store in eight still turns at least one real AI crawler away somewhere else, and many miss the basics that make their products and their identity readable by machines.

What we checked

Each site went through the same eleven checks, grouped into three areas:

• Access: does robots.txt exist and does it disallow known AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others); do both the bare domain and the www address respond; and — most importantly — can the AI services themselves open the site. For that we asked Claude, Gemini and Perplexity, through their official APIs, to fetch each home page with their own crawlers.
• Readability: how much visible text the server returns without running JavaScript.
• Structure: schema.org Product markup on a product page found via the sitemap, Organization markup on the home page, an XML sitemap, and an llms.txt file.

The sample covers 189 fashion and 133 beauty brands across luxury, premium and mass-market segments — from global houses to direct-to-consumer labels and multi-brand retailers. Full methodology and limitations are at the end of this report.

Finding 1: Almost no one blocks AI crawlers by policy

Only 9 of 319 sites (3%) disallow AI crawlers by name in robots.txt. The overwhelming majority leave AI bots under the general wildcard rule, which usually allows the catalogue. This matches other industry research showing fashion brands block AI crawlers far less often than general retail.

A further 13% of sites either had no robots.txt or did not return it. That is not a block, but it leaves each crawler to apply its own default.

Finding 2: One store in eight turns a real AI crawler away

robots.txt is only half the story. A site can allow every bot there and still refuse them at the firewall, or serve them an empty page. To see what AI services actually get, we asked Claude, Gemini and Perplexity — through their official APIs — to open each home page. The request then comes from each company's own crawler, from its own network, exactly as it would when a shopper asks about your brand.

Share of stores that refused each AI service. Perplexity returned a clear answer for 115 sites; Claude and Gemini for 317–319.

13% of stores refused at least one of the three services, and 4% refused every service that gave a clear answer. Luxury brands were the most closed: 19% refused at least one service, against 12% of premium and 11% of mass-market brands. Several global luxury and prestige beauty houses refused both Claude and Gemini.

The test also exposed a common false alarm. When we sent a request that merely claimed to be OpenAI's GPTBot from our own network, 145 sites refused it — 89% of the Shopify stores among them. Every one of those 145 sites opened normally for the real AI crawlers. Bot protection checks where a crawler really comes from, so an impostor test says nothing about real access. Any tool that measures AI access that way will report problems that do not exist.

Finding 3: One in four product pages lacks Product markup

We located a product page through the sitemap on 220 sites. Of those, 76% carried schema.org Product markup and 24% did not. Product markup is the machine-readable version of what you sell: name, brand, price, availability, images. Without it, an assistant has to infer these facts from the page layout — or from a retailer's page instead of yours.

Share of product pages with Product schema, by segment. Luxury brands lag most, often because product pages are built as editorial experiences rather than structured catalogues.

Finding 4: A third of brands do not tell machines who they are

32% of the home pages we could read had no Organization markup — the structured statement of a brand's name, logo, official social profiles and contact details. It helps assistants connect your products to the right entity and avoid confusing you with similarly named brands. European sites lagged: 60% had Organization markup, against 74% in the US and 73% in Australia.

Finding 5: Sitemaps and llms.txt are inconsistent

• 27% of sites had no XML sitemap at the standard locations or referenced in robots.txt. Crawlers can still find pages through links, but new and seasonal products are discovered more slowly.
• 49% already publish an llms.txt file — a new, optional convention that gives language models a map of the site. Adoption is highest in the US (67%) and Australia (66%), lower in Europe (51%). No major assistant has confirmed llms.txt as a ranking signal, so its absence is not a failure, but its fast spread shows how seriously brands are taking AI discovery.

Finding 6: Most sites serve real content without JavaScript

Among sites where we received a real page, 8% returned under 350 characters of visible text in the server response, and 13% returned under 1,200 — a sign that the page is assembled in the browser by scripts. Most AI crawlers read the server response and do not run scripts like a browser, so for these sites assistants may see little more than a shell.

How segments and regions compare

Scores are out of 100 and cover all 319 analysed sites; crawler access is scored from the real AI-crawler test, and response speed is excluded. Markup and llms.txt percentages use the 246 sites that served their home page to our scanner; Product schema uses the 220 sites where a product page was found.

Who does it best

The highest scores went to brands that combine open crawler rules, server-rendered pages and complete structured data:

• 100/100: Lacoste, Gap, Old Navy, Banana Republic, Swarovski, Fenty Beauty, Summer Fridays, Pat McGrath Labs, Ksubi
• 92–93/100: Anastasia Beverly Hills, Murad, Tory Burch, Madewell, American Eagle, Alo Yoga, Vuori, Spanx, Beauty of Joseon

The pattern is not size or price point. It is operational discipline: a catalogue built as structured data first, presented second.

What brands should do

1. Add Product markup to every product page — name, brand, SKU, price, availability, images, and variants. This is the single most common gap in our sample.
2. Add Organization markup to the home page with your official name, logo and social profiles.
3. Publish an XML sitemap and reference it in robots.txt.
4. Test real AI crawler access — not just robots.txt. Check your CDN or platform bot settings allow verified AI crawlers, and that the product description is in the server HTML.
5. Then work on the content itself — the facts assistants quote. See our AI-ready product page checklist.

Methodology and limitations

• Sample: 322 brand websites selected to cover fashion (189) and beauty (133) brands and retailers in the US (136), Europe (133) and Australia (53), across luxury, premium and mass-market segments. It is a structured sample of well-known brands, not a random sample of all online stores.
• Dates: checks ran on October 13, 2026.
• Tooling: the same automated site readiness checks used in the Sayrank product. robots.txt, markup, sitemap and llms.txt checks use plain HTTP requests and parsing. Crawler access was tested by asking Claude (web fetch tool, cache disabled), Gemini (URL context tool) and Perplexity (Agent API fetch tool) to open each home page; a site counts as refusing a service when that service reported it could not retrieve the page.
• Reachability: 3 sites timed out entirely. A further 73 did not serve their home page to our own scanner — but the AI services opened 62 of them, confirming that bot protection screened our scanner, not the real crawlers. These sites are included in the access and file-level figures and excluded from page-level markup figures.
• ChatGPT: OpenAI's API offers no tool that fetches a specific page with OpenAI's crawler, so ChatGPT access is covered only by the robots.txt check. Perplexity gave a clear result for 115 sites; for the rest its answer contained too little page content to judge either way.
• Product pages: one product page per site, located via the sitemap; not found on 99 sites.
• Speed: response times were measured but are not reported, because they depend heavily on the distance between our scanner and each site's servers.

You can run the same checks on your own store in Sayrank, which also tracks how ChatGPT, Claude, Gemini and Perplexity answer your shoppers' questions and drafts the fixes for what it finds.