llms.txt: what it is, and whether it does anything

llms.txt is a Markdown file you put at /llms.txt that tells a language model which pages on your site are worth reading and in what order. It was proposed by Jeremy Howard in September 2024, it has a clean spec at llmstxt.org, and thousands of sites have shipped one.

Here is the part most posts about it will not tell you: no major AI engine has publicly confirmed that it reads llms.txt, and none has confirmed using it as a ranking or citation input. Not OpenAI. Not Google. Not Anthropic. Not Perplexity. If you check your access logs for requests to /llms.txt from known AI user agents, you will usually find close to zero.

So why write about it? Because it takes about an hour to ship, it cannot hurt your rankings, and the file forces you to answer a question you should be able to answer anyway: what are the 20 pages on my site that best explain what we do? Also because the file most people conflate it with, robots.txt, absolutely does get read, and getting that one wrong will quietly delete you from AI answers.

What is in an llms.txt file?

An llms.txt file is Markdown with a fixed shape: an H1 with the site name, an optional blockquote one-liner, optional prose, then H2 sections holding bulleted lists of links. Each bullet is - [Title](url): one-line description. That is the entire spec.

The one special section name is ## Optional, which the spec defines as content a model may skip if it is short on context. Everything else is up to you.

Here is a complete, real-shaped file for a product like ours:

# Spottlo

> Spottlo tracks whether ChatGPT, Perplexity, Gemini and Google AI Overviews
> mention your brand when buyers ask questions in your category. It runs your
> prompt set weekly and reports mentions, rank position, and Share of Voice
> against named competitors.

Spottlo is an AI visibility tracker (also called a GEO tool or AI search
monitoring tool). Pricing is $39/mo base plus $19/mo per additional brand, and
all four engines are included on every plan. A free report requires no signup.

## Product

- [How Spottlo works](https://spottlo.com/how-it-works): The scan pipeline — how a prompt set is run against four engines weekly, how mentions are detected, and how rank and Share of Voice are computed.
- [Pricing](https://spottlo.com/pricing): $39/mo base, +$19/mo per extra brand, 25 tracked prompts per brand, all four engines on every plan, optional $19/mo daily-scan add-on.
- [Free AI visibility report](https://spottlo.com/): Enter a domain and get a no-signup report showing whether AI engines currently mention the brand.

## Concepts

- [AI visibility glossary](https://spottlo.com/glossary): Definitions for GEO, AEO, Share of Voice, prompt set, citation, mention rate, and related terms.
- [What is generative engine optimization?](https://spottlo.com/blog/what-is-generative-engine-optimization): The practice of getting a brand cited inside AI-generated answers, and how it differs from ranking blue links.
- [How AI engines choose sources](https://spottlo.com/blog/how-ai-engines-choose-sources): Retrieval, reranking and citation selection in ChatGPT, Perplexity, Gemini and AI Overviews.
- [AI Share of Voice, explained](https://spottlo.com/blog/ai-share-of-voice-explained): How to compute the share of AI answers in a category that mention your brand rather than a competitor.

## Guides

- [How to rank in ChatGPT](https://spottlo.com/blog/how-to-rank-in-chatgpt): What ChatGPT's browsing and memory layers actually retrieve, and how to be in the set.
- [How to appear in Google AI Overviews](https://spottlo.com/blog/how-to-appear-in-google-ai-overviews): The relationship between the classic index and the AI Overview citation set.
- [Schema markup for AI search](https://spottlo.com/blog/schema-markup-for-ai-search): Which JSON-LD types still matter for entity resolution, with copy-pasteable examples.
- [Structuring a page for AI citation](https://spottlo.com/blog/content-structure-for-ai-citation): Chunking, answer-first passages, and the lift-out test.

## Comparisons

- [All AI visibility tools compared](https://spottlo.com/compare): Side-by-side pricing and engine coverage for Profound, Peec AI, Otterly.AI, Scrunch AI, Ahrefs Brand Radar, Semrush AI Visibility Toolkit, Trakkr and AthenaHQ.
- [Spottlo vs Profound](https://spottlo.com/vs/profound): Profound starts at $99/mo but tracks ChatGPT only; three engines requires the $399/mo tier.
- [Spottlo vs Otterly.AI](https://spottlo.com/vs/otterly-ai): Otterly's $29/mo Lite plan covers 15 prompts and four engines, with Gemini and AI Mode as paid add-ons.

## Optional

- [Blog index](https://spottlo.com/blog): All posts.
- [Changelog](https://spottlo.com/changelog): Release notes.

Two things to notice. First, the descriptions do work. - [Pricing](/pricing): Pricing page is useless; the version above tells a model the actual price without it having to fetch the page. Second, the blockquote is the highest-leverage line in the file, because it is the one sentence a model is most likely to reproduce.

llms.txt vs llms-full.txt

llms.txt is an index of links. llms-full.txt is the entire body of your documentation concatenated into a single Markdown file, so a model (or a human pasting into a context window) can ingest everything in one fetch.

If you have docs, llms-full.txt is the more useful of the two, and it is the one people actually use. Generating it is a build step, not a hand-written file:

#!/usr/bin/env bash
# scripts/build-llms-full.sh — concatenate docs into /public/llms-full.txt
set -euo pipefail

OUT="public/llms-full.txt"
{
  echo "# Spottlo — full documentation"
  echo
  echo "> Generated $(date -u +%Y-%m-%d). Source: https://spottlo.com"
  echo

  # Deterministic order beats find's default order.
  find docs -name '*.md' -print0 \
    | sort -z \
    | while IFS= read -r -d '' f; do
        echo
        echo "---"
        echo
        echo "<!-- source: https://spottlo.com/${f%.md} -->"
        echo
        cat "$f"
      done
} > "$OUT"

echo "wrote $OUT ($(wc -c < "$OUT") bytes)"

Serve it with Content-Type: text/markdown; charset=utf-8. Keep it under a couple of megabytes; a 40 MB file is not going into anyone's context window.

Does llms.txt actually do anything?

There is no evidence it does, and you should not plan around it doing anything. Here is the honest scorecard as of mid-2026:

That last row is the whole argument. The expected value is small but the cost is an hour, so ship it. What you should not do is ship llms.txt, put it in a slide, and tell your team you have "done GEO." The engines that decide whether you get cited are reading your HTML, not your Markdown manifest. That is what how AI engines choose sources gets into.

Which AI crawlers should robots.txt allow?

This is the part that has verifiable consequences. If robots.txt disallows OpenAI's search crawler, you will not be cited in ChatGPT, full stop. We see this in scan data more often than you would expect: a brand with strong classic SEO shows zero AI mentions, and the cause is a Disallow: / under a bot they blocked in 2023 during the training-data panic and forgot about.

The distinction that matters is training crawlers vs search/retrieval crawlers. They are different user agents, and you can allow one and block the other.

The trap in that table is Google-Extended. Blocking it does not keep you out of AI Overviews, because AI Overviews are generated from the regular Googlebot index. There is no robots.txt token that removes you from AI Overviews while keeping you in Search. Your only lever there is nosnippet / max-snippet:0, which also nukes your regular snippet. That is the trade.

A copy-pasteable robots.txt

This is the configuration for a business that wants to be found in AI answers and does not particularly mind being trained on. If you do mind, flip the two training bots to Disallow: / and leave everything else alone.

# robots.txt — https://example.com/robots.txt
# Posture: fully open to AI search and retrieval. Training allowed.

# ---------- Classic search (also powers AI Overviews, AI Mode, Copilot) ----------
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# ---------- OpenAI ----------
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# ---------- Anthropic ----------
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# ---------- Perplexity ----------
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# ---------- Google Gemini training / grounding ----------
User-agent: Google-Extended
Allow: /

# ---------- Apple Intelligence ----------
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# ---------- Other training crawlers ----------
User-agent: Amazonbot
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: CCBot
Allow: /

# ---------- Everything else ----------
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*session=
Allow: /

Sitemap: https://example.com/sitemap.xml

Two rules people get wrong. Robots.txt matching is not case-sensitive on the user-agent token, but the token must match the bot's declared product name exactly (OAI-SearchBot, not OAI-Searchbot-1.0). And a bot obeys only the single most specific matching group — if you have a User-agent: * block with Disallow: /blog/, and a separate User-agent: GPTBot block, GPTBot ignores the wildcard block entirely. That cuts both ways: it means your named allow-blocks work, and it means a permissive wildcard will not rescue a bot you explicitly disallowed elsewhere.

Verify it, don't trust it

Config drift is real. CDNs ship bot-management rules that block AI crawlers at the edge regardless of what robots.txt says, which is a fun way to spend a Thursday. Two checks:

# 1. Does the origin serve the page to a bot user agent at all?
curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://example.com/pricing | head -1
# Want: HTTP/2 200.  A 403 means your CDN/WAF is blocking it, not robots.txt.

# 2. Are the bots actually arriving? Grep your access logs.
grep -Ei 'gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot|google-extended' \
  /var/log/nginx/access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

Cloudflare in particular has a one-click "Block AI Scrapers and Crawlers" toggle that is on by default for some plans. If that toggle is on, your robots.txt is decorative. Check it.

Why bother if the engines ignore llms.txt?

Because the marginal cost is an hour and the file has second-order uses, and because writing it surfaces a real problem: most sites cannot name their 20 best pages.

The concrete uses that exist today:

• Coding agents and doc-ingestion pipelines do look for /llms.txt and /llms-full.txt. If you sell a developer tool, someone is going to paste your llms-full.txt into a context window this week.
• It is a forcing function. Writing one-line descriptions for your key pages is the same exercise as writing good meta descriptions and good <h1>s, and it exposes the pages that have no clear job.
• It is cheap insurance. If an engine does start honoring it in 2027, you already shipped it.

What it will not do is make ChatGPT recommend you. Being recommended comes from being the answer on pages that engines can retrieve, and from being mentioned by sources the engine already trusts. Ship llms.txt in the morning, then go do that.

If you want to know whether any of this is working, you have to measure the output, not the artifacts. Spottlo runs your buyer questions through ChatGPT, Perplexity, Gemini and Google AI Overviews weekly and tells you whether your brand actually showed up. Shipping llms.txt and never checking whether mentions moved is just cargo cult with a Markdown file.

What to do next

1. Audit robots.txt first. Fetch https://yourdomain.com/robots.txt and look for Disallow: / under GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended. If any of those are blocked and you want AI visibility, fix it today. This is the only item on this list with a proven effect.
2. Check your CDN. Cloudflare, Fastly and Akamai all ship bot-management rules that can block AI crawlers above robots.txt. Confirm with the curl -A check above that a bot user agent gets a 200, not a 403.
3. Grep your access logs for AI crawler hits over the last 30 days. Zero hits from OAI-SearchBot means something is blocking it.
4. Ship llms.txt. One H1, one blockquote, three or four H2 sections, 15 to 25 links with real descriptions. Serve as text/markdown. Then stop thinking about it.
5. Measure the outcome. Run a free visibility report to see whether the engines mention you at all today, and re-run it after the crawler fixes land. If your prompt coverage is thin, how to build a prompt set is the next thing to read; if the crawlers can reach you but you are still not cited, the problem is the pages themselves, and content structure for AI citation is where to go.