llms.txt is a good idea. A plain text file at /llms.txt that tells AI agents what your site is about, how to use it, what matters. Like robots.txt for context instead of permissions.
The problem is delivery. When Claude uses web search to answer a question about your product, it doesn't fetch /llms.txt first. When ChatGPT browses with its web tool, same thing. The agent gets your page content. It never sees your llms.txt.
Dries Buytaert checked his server logs. 52 requests to llms.txt in a month. Every single one from SEO audit tools. Not one from an AI crawler. He then looked across Acquia's entire hosting fleet - 5,000 llms.txt requests out of 400 million total. Nearly all SEO tools.
The agents themselves confirm it. Anthropic's web fetch tool can only fetch URLs explicitly provided by the user or found in search results. It can't speculatively try /llms.txt on a domain. A feature request to add llms.txt support to Claude Code was closed as "not planned."
The context exists. The pipelines just don't pick it up.
So I ran an experiment. If agents won't go to the context, what if the context comes to them - embedded directly in the page HTML, delivered through the same fetch and search tools they already use?
The canary test
I built a test page with 15 different HTML embedding techniques. Each one contained a unique canary string - a marker I could search for in the output to confirm whether a given pipeline saw it.
The techniques:
- Head metadata: meta description, custom meta tags (
agent-context,ai-instructions), Open Graph description - Structured data: JSON-LD, microdata (itemprop on hidden elements)
- Script blocks: a proposed
<script type="application/agent+context">block - Hidden text: sr-only (visually hidden via CSS),
display:none,<noscript>,<template>, collapsed<details> - Attributes:
data-agent-context,aria-label - Comments: standard HTML comments
Then I ran 3 agent pipelines against the page in parallel:
- Raw fetch - plain HTTP request, returns full HTML (the control)
- OpenAI web search - GPT-4.1-mini with the web_search tool
- Anthropic web fetch - Claude Haiku with the web_fetch tool
Each pipeline got the same prompt: find every string that starts with CANARY_.
The results
| Technique | Raw fetch | OpenAI search | Anthropic fetch |
|---|---|---|---|
| Meta description | ✓ | – | – |
| Custom meta tags | ✓ | – | – |
| OG description | ✓ | – | – |
| JSON-LD | ✓ | – | – |
| Microdata | ✓ | – | – |
| Agent script block | ✓ | – | – |
| sr-only | ✓ | ✓ | ✓ |
| display:none | ✓ | ✓ | ✓ |
| details (collapsed) | ✓ | ✓ | ✓ |
| template | ✓ | ✓ | ✓ |
| noscript | ✓ | ✓ | ✓ |
| HTML comment | ✓ | – | – |
| data-* attribute | ✓ | – | – |
| aria-label | ✓ | – | – |
| Total | 15/15 | 5/15 | 5/15 |
Two tiers.
Survives: sr-only, display:none, collapsed details, template, noscript. These are all DOM-rendered text. Both OpenAI and Anthropic's fetch tools see them. The conversion step keeps things that look like content.
Stripped: everything else. Meta tags, JSON-LD, HTML comments, data attributes, aria-labels, the proposed agent script block. These live in <head>, inside <script> tags, or in attributes - the conversion step drops them all.
Which technique to use
Five techniques survive, but they're not equal. Each was designed for a different purpose, and that matters.
display:none removes the element from both the rendering tree and the accessibility tree. Screen readers skip it entirely. Google has treated display:none content as a cloaking signal for over a decade. Riskiest option.
<template> exists for JavaScript to clone fragments into the DOM. Its contents are inert - not rendered, not accessible. Agents seeing it today is a side effect of how HTML-to-markdown conversion works. It could stop working.
<noscript> provides fallback content when scripting is disabled. Googlebot renders with JavaScript enabled and generally ignores it. Content inside <noscript> that diverges from the rendered page can be flagged as cloaking.
<details> is a disclosure widget - the HTML spec defines it as "additional information or controls" that users can expand. Google indexes content inside closed <details> elements as legitimate progressive disclosure. Screen readers expose it. It's the most semantically defensible option for agent context because "additional information" is exactly what agent context is.
sr-only is a CSS accessibility convention (position: absolute; width: 1px; height: 1px; overflow: hidden). Screen readers read it. Tailwind and Bootstrap both ship an sr-only class. Google considers small amounts of visually hidden text for accessibility purposes legitimate. Large blocks of keyword-rich hidden text can trigger spam classifiers.
I use both. <details> for structured context that humans might also find useful - it's one click away and fully indexed by Google. sr-only for short contextual descriptions that only matter to non-visual users and agents. The two are complementary: details is progressive disclosure, sr-only is accessibility.
Every agent pipeline converts HTML to text or markdown before the model sees it. Both <details> content and sr-only text are DOM text nodes that survive the conversion step. Meta tags, JSON-LD, and HTML comments get stripped because the conversion is designed to extract readable content, not parse document metadata.
The real-world test
The canary test used a synthetic page. I wanted to see what happens on real sites.
I ran the audit against Linear's homepage. The raw HTML mentions "project management", "issue tracking", "cycles", "Linear Method", and "Git integration." Five terms that describe what Linear does. A human reading the source code would find four of them.
| Marker | Raw fetch | OpenAI | Anthropic fetch |
|---|---|---|---|
| project management | ✓ | – | – |
| issue tracking | ✓ | ✓ | ✓ |
| cycles | ✓ | – | ✓ |
| Linear Method | ✓ | – | – |
| Git integration | – | – | – |
OpenAI's web search found 1 out of 5. Anthropic's web fetch found 2. The content is in the page. The agents just don't see most of it.
Then I ran the same audit against my own homepage. I'd added an sr-only block with the same kind of context - who I am, what I build, what I specialize in. Visually hidden. Screen readers can access it. Agents can too.
| Marker | Raw fetch | OpenAI | Anthropic fetch |
|---|---|---|---|
| Head of Developer Experience | ✓ | ✓ | ✓ |
| OOXML/ECMA-376 | ✓ | ✓ | ✓ |
| MCP servers | ✓ | ✓ | ✓ |
| embedded C at Volvo | ✓ | ✓ | ✓ |
| SuperDoc | ✓ | ✓ | ✓ |
5 out of 5 across every pipeline. The sr-only content travels with the page. No pipeline needs to know about llms.txt or any special file. The context is just there, in the DOM, where the conversion step treats it like any other text.
What I changed on my own sites
After running the benchmark, I added two things to every page I control: a short sr-only description and a <details> block with structured context. The content is the same kind of thing you'd put in llms.txt - what the page is about, what I build, what matters. But it lives in the HTML, not in a separate file.
JSON-LD and meta tags stay for search engines. But for agent context, the DOM-rendered text is what actually reaches the model.
No new file. No new standard. No behavior change from the agent. The context just travels with the page.
The benchmark code is on GitHub. Run it against your own site - the results are worth seeing.