Content and reviews get most of the attention in AI search conversations, but a technical layer underneath decides whether AI systems can read your site at all, and how accurately they interpret what they find. This post covers the three pieces of that layer: the emerging llms.txt convention, the schema types that matter most for AI interpretation, and the crawler permissions that gate everything else.
What is llms.txt?
llms.txt is a proposed convention for a plain-text markdown file placed at the root of your website, at yoursite.com/llms.txt. Its purpose is to give large language models a curated, token-efficient map of your site: what the business is, which pages matter most, and where to find authoritative details.
The reasoning behind it is practical. When an AI system visits a typical business website, it encounters navigation menus, cookie banners, scripts, and layout markup surrounding a small core of useful text. An llms.txt file strips all of that away and hands the model a clean summary with links to your most important pages.
Two honest caveats. First, llms.txt is a community proposal, not an accepted standard, and major AI companies have not committed to consuming it. Second, it supplements rather than replaces normal crawlability. Treat it as a low-cost, low-risk addition: it takes minutes to create, cannot hurt, and positions you well if adoption grows.
What does an llms.txt file look like?
The format is deliberately simple markdown: an H1 with the site name, a blockquote summary, then sections of annotated links. A trimmed example for a plumbing company:
# Coastal Flow Plumbing
> Licensed plumbing and drainage company serving New Westminster,
> Burnaby, and Coquitlam, BC. Residential service, repair, and
> installation since 2012.
## Services
- [Drain cleaning](https://example.com/drain-cleaning): Hydro
jetting and camera inspection for residential drains
- [Water heaters](https://example.com/water-heaters): Tank and
tankless installation and repair
## Company
- [About](https://example.com/about): Licensing, team, and
service area details
- [Contact](https://example.com/contact): Phone, address, hours
Keep it short, factual, and current. List only pages that genuinely define your business, and phrase each annotation as a plain statement a machine could quote. Update it when your services or service area change, exactly as you would a sitemap.
Which schema types matter most for AI search?
Schema markup, written as JSON-LD in your page code, converts human-readable content into explicit machine-readable claims. Search engines have consumed it for years, and it serves AI systems the same way: as disambiguation. Five types earn their keep for a local business.
Organization. Sitewide identity: legal name, logo, URL, phone, and sameAs links to your social and directory profiles. The sameAs property is quietly powerful because it explicitly connects your scattered web presence into one entity.
LocalBusiness (or a specific subtype like Plumber, Dentist, or HVACBusiness). Extends Organization with address, geo coordinates, opening hours, price range, and areaServed. Use the most specific subtype that applies; a dental practice should be Dentist, not generic LocalBusiness.
Service. Describes each individual service with its own name, description, provider, and areaServed. Most local sites skip this, which makes it an easy differentiator.
FAQPage. Marks up question-and-answer pairs so machines can extract them verbatim. Apply it to genuine Q&A content, since the questions and answers must be visible on the page.
BreadcrumbList. Encodes your site hierarchy, helping crawlers understand how pages relate, which pages are parents, and where a given service sits in your structure.
Reference documentation for all of these lives at developers.google.com, and the details of implementation belong in the same bucket as the rest of your technical SEO.
How do I configure robots.txt for AI crawlers?
AI platforms identify themselves with named user agents, and your robots.txt controls their access. The agents worth knowing:
- GPTBot: OpenAI's crawler for model training data
- OAI-SearchBot: OpenAI's crawler for ChatGPT search results
- ClaudeBot: Anthropic's crawler
- PerplexityBot: Perplexity's index crawler
- Google-Extended: a control token governing whether Google may use your content for AI model training; it does not affect normal Googlebot indexing or your search rankings
- Bingbot: Microsoft's standard crawler, relevant because ChatGPT retrieves web results through Bing
For a business that wants AI visibility, the sensible default is to allow all of these. The common failure mode is accidental blocking: security plugins, CDN bot-management rules, and copied boilerplate robots.txt files sometimes deny AI agents wholesale. Audit your live file at yoursite.com/robots.txt and your CDN's bot settings, not just your CMS configuration, because blocks can be applied at either layer.
A minimal explicit-allow block looks like this:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
How do I verify all of this is working?
Deployment without verification is guesswork. Run through this sequence after any changes.
- Validate schema. Paste each key page into Google's Rich Results Test or the Schema.org validator and confirm your JSON-LD parses without errors and the values match the visible page.
- Confirm crawler access. Fetch yoursite.com/robots.txt directly in a browser and read what is actually served. If you use Cloudflare or similar, check its bot-fight settings separately.
- Check your server logs or CDN analytics. Look for visits from GPTBot, ClaudeBot, and PerplexityBot user agents. Their presence confirms the pipeline works end to end.
- Test rendering. View source on your important pages and confirm the main content and JSON-LD appear in the raw HTML, not only after JavaScript executes. Crawlers vary in their rendering capability, and content present in initial HTML is the safe baseline.
- Ask the assistants. Periodically ask ChatGPT, Claude, and Perplexity what they know about your business and whether they can summarize your services page. Wrong answers often trace back to a technical gap or stale third-party data you can then correct.
Where does this fit in the bigger AI search picture?
Technical readiness is the foundation, not the whole building. Clean crawler access and accurate schema make your business legible to machines, but legibility only pays off when there is substance worth reading: service pages that answer real questions, reviews that corroborate your claims, and third-party mentions that establish trust.
Sequence the work accordingly. Fix crawler blocks first because they gate everything downstream. Add Organization and LocalBusiness schema next because entity clarity underpins every AI mention. Then shift your energy to content and reputation, where the compounding returns live. The full strategic context is laid out in our AI search optimization guide, and if your website's platform makes schema work painful, our web design service builds these foundations in from the start.
Frequently asked questions
Is llms.txt required for AI search visibility?
No. It is an optional, emerging convention that major AI platforms have not formally adopted. It costs little to implement and may help AI systems summarize your site efficiently, but crawlable pages, schema markup, and consistent business information matter far more today.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content may be used for training Google's AI models. It is separate from Googlebot, so blocking it does not remove you from search results, though it may limit how Google's AI products use your content.
Where should schema markup go on my site?
JSON-LD belongs in the HTML of the relevant pages: Organization and LocalBusiness sitewide or on your homepage and contact page, Service schema on each service page, FAQPage on pages with visible Q&A content, and BreadcrumbList wherever breadcrumbs appear. Most modern platforms and SEO plugins can inject it, but always validate the output.
How do I know if AI crawlers are visiting my site?
Check your server logs or CDN analytics for user agents like GPTBot, ClaudeBot, and PerplexityBot. Regular visits confirm your robots.txt permits them and your pages are reachable, while their complete absence usually points to a block at the CDN or robots.txt layer.
Technical AI readiness is one of those jobs that is invisible when done right and costly when ignored. If you want a qualified set of eyes on your schema, crawler access, and site structure, request a free SEO audit and we will show you exactly what the machines see.