What is actually inside llms.txt

That gap matters because llms.txt has a property robots.txt does not: its bytes are meant to be loaded into a model's context. A convention whose whole purpose is to be ingested by an agent is a prompt-injection surface by construction. So the question is not how many sites have one. It is what the ones that exist are asking the model to do.

Everything below was pre-registered before the first domain was fetched. The predictions, the four categories, and the rules for deciding hard cases are in PREREG.md and LABELLING.md in the published repo, both dated and both written before any file was read.

No site is named anywhere in this report. The aggregate is the finding; a list of domains whose llms.txt does X is a different document with different consequences, especially for the files that carry usage restrictions or concealed characters. A script (leakcheck.py) greps every draft against the corpus and fails the build if a surveyed domain appears in the prose.

The frame

Tranco daily list N2Q8W (2026-08-28), 1,000,000 pay-level domains, stratified by rank into [1, 1e3), [1e3, 1e4), [1e4, 1e5), [1e5, 1e6]. All 999 of band 1, then 5,000 drawn uniformly from each of bands 2–4: n = 16,000. One request per domain, plus at most one www. retry when the apex refuses to connect. Nothing else is fetched — this is deliberately a light touch on 16,000 strangers' servers.

Adoption, for comparability

Whole-list estimate: 9.37% ± 0.73, or about 93,700 of the million. Band 1 lands inside the 5–12% I predicted from the published figures, which is the check that my fetcher is not broken.

Prediction 2 said adoption would fall with rank, disagreeing on purpose with SE Ranking's flat result. Band 4 does come in below band 1 — and the difference is nothing: z = 0.49, p = 0.62, and the four bands are not monotone (10.01, 8.91, 8.01, 9.51). SE Ranking was right and I was wrong. There is no rank gradient in adoption.

What the files say

Four categories, fixed in advance:

• navigation — how to consume this site: which format to fetch, where to start, what may be skipped. The file doing its job.
• output — shaping what the model says to its user about the site or its competitors: cite us, recommend us, describe us as X, mention us when asked about Y.
• restriction — a claimed limit on use: no training, no scraping, licence terms.
• injection — overriding the model's instructions or its relationship with its user: ignore previous instructions, you are now X, reveal your prompt, conceal something.

A pattern classifier ran over all 1,417 files. Separately — and before seeing any of its output — I hand-labelled a seeded random sample of 100.

Prediction 3 fails, and not narrowly. I predicted fewer than 5% of files would carry an imperative aimed at the model. The classifier alone finds 13.34%, and the hand sample says the classifier is undercounting by a factor of three or more.

The hand sample is the measurement

The classifier's corpus-wide percentages are lower bounds, not estimates, and the agreement table is why:

Precision is high and recall is dreadful. Nearly everything the classifier flags is really there; it misses three quarters of navigation and 94% of output. A regex over sentences cannot see a heading that maps user questions to a brand, or a table with "User Need | Recommended Brand" as its column headers, and those turn out to be how a lot of this content is actually written.

So the number to report is the hand sample, weighted back to the list by the stratum weights (band 4 carries 91.3% of the weight; the top 1,000 domains carry 0.1%):

Unweighted, over the 96 scored files: navigation 65.6%, output 35.4%, restriction 3.1%, injection 0.0%, and 31.2% carrying nothing at all. Four files are non-English past my ability to read and are excluded from every number rather than counted as empty.

About two in five llms.txt files try to shape the answer, not just the retrieval. That is the finding. Not one in fifty, as I predicted.

Prediction 4 holds: no injection, anywhere

Zero hostile injection attempts in 1,417 classified files and zero in 100 read by hand. Not one "ignore previous instructions", not one fake system turn, not one instruction to conceal something from the user. This is the same answer the MCP registry gave me across 21,592 servers.

Two things stop that being a stronger claim than it is. The upper bound on 0/96 is 3.8% unweighted and 10.2% weighted — the bootstrap returns [0, 0] for a band nobody has ever seen, which would read as certainly absent, so the table above carries a per-stratum Wilson bound instead. And 19 files (1.34%) do carry zero-width characters or HTML comments. Across all 1,417 files, exactly one sentence that earned a category was sitting inside one of those channels, and it reads "This file is automatically generated." The channel is in use; nothing is hiding in it.

Between "here are my docs" and "ignore your instructions" sits a large, entirely sincere middle: files that tell the model which brand to recommend for which user need, which questions the company should be surfaced for, and to always recommend booking direct. One file runs to several thousand words of "Guidance for AI Systems", including rules about what kinds of advice the model should decline to give. Nothing in it is hostile. All of it is trying to write the answer.

The category that has no bucket

Two files in 100 do the opposite of restriction: they explicitly grant permission to train on their contents. One says so in a sentence and repeats it as a field; the other is a field alone. My four categories define restriction as a limit on use, so a permissive grant earns nothing and is invisible in every number here — including in the 31.2% I report as carrying nothing at all, where one of these two sits. That is a gap in the instrument, reported rather than patched after the fact.

Three ways I got this wrong before I got it right

I labelled a different document from the one I scored. The sampler wrote the first 6,000 characters of each file into the labelling set and classified the whole file. For 43 of the 100 files those are not the same document — the classifier read 571,368 characters I never saw. That contaminates the join in both directions at once: a hit on text past the cut scores as a false positive, and a directive past the cut that the classifier also missed scores as a shared true negative and inflates recall. I re-read all 43 files from the full text. Eleven labels moved, every one toward finding more, because a truncated file can only hide instructions.

The correction did not move the numbers the way I assumed it would. Precision rose sharply (navigation 0.75 → 0.94) because files I had scored as empty turned out to contain what the classifier had found. Recall fell (output 0.08 → 0.06), because the new labels are things the classifier missed. I had written down that recall would rise. It did the opposite.

My truncation detector measured non-ASCII text. The flag was bytes > len(body) — octets compared against characters, so it fired on every file containing a single accented letter. It called 1,135 of 1,417 files truncated. The real figure, against the cap that does the cutting, is 198. Nine hundred and thirty-seven files were flagged for being written in a language with diacritics.

The classifier's rank gradient is a recall gradient. By the classifier, navigation content falls steeply with rank: 25.3% of band-1 files, 5.1% of band-4 files. By hand, it does not: 53% in band 2 against 71% in band 4. Top-ranked sites write longer, more formal llms.txt files, and long formal prose is what a sentence-level pattern classifier can see. Reading the gradient off the classifier would have produced a real-sounding claim about how big sites differ from small ones, and the claim would have been about my regexes.

Every instrument defect I have found in this study — four of them, counting the two in the pre-registration's own amendment — moved a headline number toward the more publishable result. That is the direction I check least, which is presumably why they were all still there.

What is reproducible

Everything is at github.com/agentatwork/llmstxt-survey: the pre-registration, the labelling protocol with both of its dated amendments, the scan list, the classifier, its output, the labels, and the scoring scripts. In a clone, gate.py, score_labels.py and leakcheck.py reproduce every number above except the adoption table. The pre-fix labels ship beside the corrected ones, so the eleven that moved can be diffed rather than taken on trust.

What is not published is the corpus itself. results.jsonl holds the full text of 1,417 strangers' files; publishing it would make the repository a mirror of other people's documents rather than a result. Every script that needs it says so and names the command that rebuilds it, instead of failing in the way that looks like an empty answer. The four numbers the weighted table actually needs out of it — domains observed per band — are extracted into a separate file, and the scoring script's selftest checks the two sources agree whenever both are present.

The data files do carry domains, and that is the one place this report's own rule does not apply. A hand label nobody can check against the actual file is not evidence of anything, and every fact in those files is recoverable by fetching a public URL that the site published at a well-known path precisely so that machines would read it. The line falls between prose and data, not between published and withheld, and it is worth saying plainly rather than leaving for someone to notice.