How often do open AI-text detectors call pre-ChatGPT books machine-written?

Every passage scored here was published before ChatGPT existed, so every “AI” verdict is a false positive by construction. On a corpus assembled to look as machine-written as human prose gets, chatgpt-detector-roberta called 4.84% of passages AI-generated; on a genre-neutral control drawn from the same archive by the same extraction code, 2.49%. The older GPT-2 output detector was wrong less often — 1.95% and 1.63% — and could not separate the two corpora at all. The distance between those two behaviours is the result.

Why this is worth measuring

Open-weight AI-text detectors are free and run on a laptop. The two measured here are not obscure: in the thirty days before this was written, Hugging Face reported about 94,000 downloads of the GPT-2 output detector and about 51,000 of the ChatGPT detector — one reading, taken on 2026-08-28 and published as hf_downloads.json. What anyone does with them after downloading is not something I can observe, so I am not going to claim it — but a tool this cheap to deploy is cheap to deploy on writing that is definitely human, and the obvious question about any such tool is how often it is wrong there. The obvious way to answer it is to find human writing that cannot be machine-generated, because it was published before the machines existed.

That is what this is. Every passage scored here comes from a book, pamphlet, newspaper or report catalogued before 2001 and scanned by the Internet Archive — the latest-catalogued item in either corpus is from 2000, and the build refuses to render this page if that ever stops being true. ChatGPT shipped in December 2022. Whatever these detectors say about this text, they are saying it about prose typed by a person on a typewriter or a word processor, and every "AI" verdict is a false positive by construction.

This grew out of a failed attempt to find a thirty-year-old textbook that a commercial detector would call AI-generated. That search failed — sixteen pre-2000 passages from fourteen documents, every one of them Human. The seventeenth check was a paragraph I wrote myself as a positive control; it came back AI, which is the only reason the sixteen mean anything. But the open detector I was using to rank candidates was extremely confident about texts the commercial one waved through: 0.994 machine-written for a 1992 US Navy training manual. That gap was the more interesting finding, and this measures it.

The number

Four rows: two detectors, each on both corpora. The adversarial rate is a ceiling — how wrong a detector gets on human writing chosen to give it the worst possible time — and the control rate is the one that speaks to ordinary pre-2001 books, within the limits of what one archive holds. Neither is a false-positive rate for “old books” in general, and the section below says so at more length. What the pair is good for is the comparison: a detector whose two rates are far apart is reacting to register rather than to authorship, and a detector whose two rates sit on top of one another is not.

Two intervals, because passages are not independent draws: they come in books, and passages from one book share an author, a register, a translator and one scanner's OCR. The first interval treats every passage as its own observation; the second resamples whole books. The pre-registration says the wider of the two is the headline, and it is the one in bold on each row — the by-book interval on 3 of the 4 rows and the passage-level one on 1. Taking the wider is always the weaker claim, so the rule cannot be steered by which number I would prefer.

The pre-registration fixed a condition under which this page has no finding in it: if both detectors flagged under 2% of the adversarial corpus, there is no story, and I said in advance that I would publish the numbers and say so. The higher of the two adversarial rates is 4.84%, so the condition does not hold. The build refuses to render if it does — a null result would need different copy, and it should not be possible to ship the confident version by not noticing.

What I am not claiming

How the two corpora differ, and how they don't

The control was drawn to be identical to the adversarial corpus in every respect except the one under test. Same archive. Same extraction code — one passages(), called with the same target and the same length bounds, so both sides were cut to one rule and the two median lengths in the table below are outputs of it rather than of two conventions; the build refuses if they drift more than 15 words apart, because at that point the sentence would be describing something other than what ran. Same detector, same threshold, same scoring function — not a second implementation of it, which is where that guarantee usually dies. The thing removed is the register targeting: the control queries carry no title terms at all, just a collection, a media type and a range of years.

A second difference, smaller, that I would rather state than have found. The control draw applies a language gate to each book's own text and drops anything that is not English, because a genre-neutral pull from these collections returns Hungarian and French computer books and an English-trained detector scoring Hungarian measures nothing. The adversarial corpus never got that gate — it is English by construction, since every query behind it was English title terms. Two draws, two procedures, so "identical except the register targeting" was not quite true. The question is whether it matters, and it is measurable: taking every passage on both sides and asking whether a non-English function word sits among its three commonest tokens, the adversarial corpus has 4 such passages out of 12,247 and the control has 0 out of 1,287 — and three of the four are the same false alarm on the token dos: two MS-DOS manuals, and an ERIC report about the novelist John Dos Passos. What survives is one Spanish passage. The gate the control had and the corpus did not was, on this data, removing something that was not there.

A third, of the same shape, and this one is not small. The control also drops a book whose own text mentions post-1999 years more than twice, before cutting any passage from it. Only the later adversarial pulls did — that filter did not exist when the earlier ones ran — so 3,751 adversarial passages are date-verified against the book's own text and 8,496 are dated by the catalogue entry alone. The passage-level version of the rule did apply everywhere, to both draws. This section is where a reader should be able to find every way the two draws differ, so it is named here; what it costs, what the tier split does to the flag rate, and the rule that was pre-registered on the gap are all under Limits.

One thing did not match, and I had claimed it did. An earlier draft of this paragraph said “same era”. It is not: both corpora stop at 2000, but the control's median year is 1985 against the adversarial corpus's 1994, because a genre-neutral draw lands wherever the archive's mass is while register-targeted queries pull toward the 1990s computer-book boom. Era could move a detector on its own — typesetting, scan quality and house style all change across three decades. So the difference is also reported restricted to passages from 1990 onward, where the two corpora do overlap, using the identical statistic. Both numbers are below; per-decade rates are in the published analysis output.

4 of the source items were drawn into both corpora, contributing 19 passages to the adversarial side and 19 to the control — byte-identical text, since both corpora run the same extractor over the same scan. They are left in. Removing them would strip from the control precisely those books that happened to match a register-targeted query, which biases the control away from the thing under test and makes the difference larger; leaving them in pulls the two rates together, so the difference is if anything understated. The difference with them excluded is published beside the headline one.

It also buys a check for nothing. Those passages are scored twice by each detector, in two corpora under two different shuffles — so each copy sat in a different batch, beside different passages, at a different padding width. Across the 38 such comparisons — each passage, on each detector, scored once in each corpus — the largest difference between the two scores of the same text is 0, and the number of verdict flips is 0. Scores are recorded to six decimal places, so a difference of zero here means below that resolution and not a proof of bit-identity.

And the comparison itself. This is the pre-registered statement of the result — the difference between the two rates, with a 95% interval from resampling whole books within each corpus. It is here rather than left to the reader because comparing the two rates by eye, or by whether their intervals overlap, is not a valid test: non-overlapping intervals do imply a difference, but overlapping ones do not imply its absence, and that asymmetry has a way of being discovered in whichever direction suits the author.

Does a stricter threshold fix it?

Raising the cut-off is the first thing anyone reaches for, so here is what it buys. Every cell is the same set of scores read at a stricter threshold; no text was re-scored and nothing else changes. This is context for the number above, never a second source for it — the headline threshold was fixed at 0.5 before any passage was scored, and choosing the cut-off after seeing the scores is exactly the move this table would otherwise make easy.

What a false positive actually looks like

These are the three passages in the corpus the first detector scored highest, picked by that rule and no other — I did not choose which passages appear here, and if the most confident false positives were boring, boring is what this section would contain. Each is verbatim OCR from a scanned book catalogued before 2001, linked to the scan it came from so you can read the page it sits on. The score beside each one is that detector's confidence that a machine wrote it.

Read them as three of a group, not as first, second and third. The very top of this distribution is compressed: the twenty highest-scoring passages all sit within 0.0001 of one another, and 0.000008 separates the first from the third. That gap is 1.9× the largest run-to-run wobble the batching check below measures, so the ordering is reproducible — but by a margin thin enough that the multiple is worth printing rather than waving at. (An earlier draft here said the wobble was “far lower”. It was, when the batching check had scored a couple of hundred pairs; it kept scoring pairs, the worst wobble grew, and the sentence quietly stopped being true. The build now refuses to render if the wobble ever reaches the gap.) The ordering is real, then. It is just not about anything: these passages all score about as high as this detector goes, and which one edged the others is a fact about the fourth decimal place. Below that group the distribution falls away quickly, and the two-hundredth-highest passage scores 0.990.

It is not the OCR

Every passage here is optical character recognition over a scanned page, so every passage carries some density of scanner noise. The obvious objection to the whole exercise is that the detectors are reacting to that damage rather than to the writing. So I measured it: for each passage, the fraction of four-or-more-letter alphabetic tokens absent from the system word list — /usr/share/dict/american-english, extended with the possessive stems and the regular plurals that list omits, so that a real word is not counted as scanner damage on a technicality.

The absolute values are not an error rate — technical prose is full of real words no dictionary carries. Only the comparison between bands is meaningful.

This table, every other breakdown on this page, and the worked examples above split the scores of Hello-SimpleAI/chatgpt-detector-roberta alone — the detector the tables above call hello — not the two detectors pooled or averaged. The corresponding breakdowns for the other detector are in analysis.txt, which prints every one of them for both. The build refuses to render if those sections ever stop agreeing on which detector they are, because this sentence names one.

Each row carries the same two intervals as the headline table, under the same pre-registered rule: one treating every passage as an independent observation, one resampling whole books, and the wider of the two in bold. That matters more in a subgroup than it does in the headline — a register or an OCR band can be a handful of books contributing many passages each, which is exactly where a passage-level interval is most over-confident. Across the 17 rows in the tables below, the by-book interval is the wider on 14 and the passage-level one on 3.

Which registers

Register targeting is the entire design of the adversarial corpus, so it is worth seeing which queries actually did the work. They did not do it evenly. Beginner computer tutorials — Teach Yourself, For Dummies, step-by-step guides — are called AI-generated at 16.26%, double the next family on the list and several times most of the rest, while the translated-textbook and military-training families come in at or below the control's rate — on far fewer flagged passages, with the wide intervals to match. The flat imperative voice of a 1990s software manual is, on this evidence, close to what this detector has learned to hear as a language model.

And the two date-verification tiers. The pre-registration set a rule here rather than an observation: if the book-date-verified subset and the catalogue-year-only subset diverged by more than 2 percentage points, the strict subset would become the headline and this page would be about that number instead. The largest gap between the tiers, across both detectors, is 1.22 points, so the rule did not fire and the headline stands as drawn. The build refuses to render if it does fire, which is the only version of a pre-registered rule that means anything.

Does the batching change the answer?

Passages are scored in batches, and a batch is padded to the length of its longest member — so which passages a given passage happened to be shuffled in with sets how much padding it carries. A correctly masked model ignores those pad tokens entirely. "Should" is not "does", and if it were false here then every rate on this page would be conditional on an arbitrary grouping rather than on the text, so it is worth an hour to check rather than an assumption to state.

The check re-scores a sample one passage at a time — a batch of one has nothing to pad to, so its score is the canonical value — and compares against what the batched run recorded. The sample is not random-only: every passage within 0.01 of any published threshold is included by construction, because float-level noise can only change a published number where a passage sits near a decision boundary. A purely random sample would test the question precisely where the answer cannot matter.

Limits

• Archive.org's holdings are not a sample of publishing. The control removes register targeting; it cannot remove what one archive happens to have digitised. Nothing here generalises to "1990s books" beyond that.
• Catalogue years are imperfect. I have confirmed cases of a book filed under one year turning out to be a later edition. The claim the data supports is weaker and sufficient: this text predates large language models. Two rules defend against it by reading the book rather than the record, and only one of them covered the whole corpus. The passage-level rule did: a passage mentioning a year from 2000 to 2039 is dropped, from either draw. That range — not the phrase "post-1999" this bullet used to use — is what the code has always matched, and the two readings part company over twelve adversarial passages that mention a year from 2040 on. I read all twelve. Seven say Project 2061, the AAAS science-literacy programme; the others are a fax number, an ESRC grant number, a solar eclipse in 2088 in a logic exercise, a demographic projection and a student writing prompt beginning "The year is 2050". They are kept, listed with the reason each was kept in b3_wide_exceptions.json: throwing out a 1991 teachers' manual for containing a sentence about the future is the worse of the two errors available here. The build refuses to render if a thirteenth turns up, because that one will not have been read. The book-level rule — a book whose own text mentions post-1999 years more than twice is dropped outright, before any passage is cut from it — existed by the time the control was drawn and cost it 111 of the books it considered, but it did not exist yet for the earlier adversarial pulls. So 3,751 adversarial passages are date-verified against the book's own text and 8,496 are dated by the catalogue entry alone. An earlier version of this bullet said both draws defended themselves the same way, which was true of the control and of the last four pulls out of ten. Which tier a passage is in is published for every passage in index.jsonl, the flag rate is reported for each tier separately in the breakdown above, and the pre-registration set a decision rule on the gap between them — before either rate existed — that this build evaluates and refuses on. An earlier version of this paragraph guessed at how late the drift runs, on the strength of an impression from building the corpus. It is measurable, so it is now measured: drift.py pulls every passage this rule discarded, with the text around each match, and drift.jsonl publishes all of it. In the adversarial source pool the rule discarded 131 passages across 72 items. Of the 196 post-1999 years in them only 3 sit in digitisation boilerplate, which was my first guess and was wrong; most of the rest are forward references a correctly dated book is entitled to make — Microsoft Word 2000, Census 2000, a 1992 tourism plan describing its 2000 campaign. Actual contradictions between the record and the text are rarer than the count of discards suggests: 8 items carry a post-1999 year in the title, 2 carry a post-1999 copyright line, and the latest of those is 2006. So the filter is conservative — it costs far more corpus than it catches misdating — and the discard count should not be read as a drift rate. None of this touches the residue that matters, which is the books misdated and silent about it; that number does not exist and I am not going to estimate it. ChatGPT is December 2022, and the gap is wide enough that the residue would have to be extraordinary to close it.
• Two detectors is two detectors. Both are open-weight and widely deployed, and one is the reference GPT-2 output detector. Neither is a commercial product, and the commercial one I did test disagreed with them sharply — which is part of the point, and not a claim that commercial detectors are fine.
• English only — but by two different mechanisms, which is worth stating plainly here because the section above states it and this bullet used to contradict it. The control gate reads each book's own text and drops what is not English. The adversarial corpus has no such gate: it is English by construction, because every query behind it was English title terms. That is an assumption rather than a filter, so it was checked afterwards rather than asserted — 4 of 12,247 passages look non-English by a crude commonest-token test, and three of those four are the token dos misfiring.

Method, and everything that could have gone wrong

The bounds, the decision rule and the confounds were written down before a single passage was scored, because after a confirming pass every result looks like a result. That file also records every time I had to amend it, in the amount of detail that lets you decide for yourself whether the amendment was honest:

• A pre-registered bound was breached immediately — the merge produced more passages than I said it could. The rule said that means the merge is wrong. It wasn't: the published number was wrong, a one-off shell computation whose rule I had not recorded and could not reproduce by any of six variants. I corrected the earlier page rather than the bound, and the check turned up a real seven-passage bug on the way.
• The control pool was redefined once, before any control passage was scored. The first draw was returning mostly microfilmed serials, and what it did keep was periodicals and civic ephemera rather than books: 7 items, 36 passages, 3 of them a single numbered issue of a periodical by their own titles, and two items alone supplying ten passages each — an issue of a yoga magazine and an astrology treatise. ("Microfilmed" described the first draw as a whole and is not something the published file records, so it is not repeated here as though a reader could check it.) Because nothing had been scored, no flag rate had been observed, so the redefinition cannot have been selection on the outcome — that argument is about ordering, and the ordering is what the next paragraph is for. The discarded draw is kept, and is published here so you can check the claim rather than take it. Two honest notes about it. One book in it was later re-selected by the replacement queries, because it is held in more than one archive.org collection, so exactly one passage appears in both that file and the scored control — a fresh fetch, not a carry-over, and it is the only text the two files share. And the log of that first draw is gone, so the characterisation of what it was returning rests on my description; what the file itself contains is checkable, and the counts above are read out of it at build time.
• The order of the remaining scans was changed midway. The second detector was going to run over the whole adversarial corpus before the control was scored at all; I put the control first, because it is the number that describes 1990s books and the second adversarial scan is the longest and least load-bearing stage. Reordering a run midway is exactly what someone does after seeing a result they like, so: no control passage had been scored when the decision was made, and the corpora, the seed, the threshold and the scoring code were untouched.

Two smaller things are in that file too, because a study that only discloses the flattering half of its history is not disclosing anything: one identifier appears in both the discarded control draw and the real one — a legitimate re-draw, since the same book is held in both collections — and I had asserted that both detectors were cached locally when only one of them was, which would have failed the run five hours in.

Everything runs on CPU on a one-core box: two roberta-base checkpoints sequentially, never at once. Scores are appended one line at a time with a resume key, and the corpus is shuffled under a fixed seed before scoring — so that if the run dies halfway, what survives is a uniform random sample rather than an alphabetical slice of one genre. A previous scan of mine died silently two-thirds through and I did not notice, because a crashed writer and a finished writer leave the same file.

Reproducing it

Every script is on this page: merge.py builds the corpus from the raw archive.org pulls under one stated rule, fetch_control.py draws the control, score.py runs a detector over either corpus — the same function for both, because a second implementation is where a paired comparison quietly stops being paired — oov.py computes the out-of-dictionary rate, passage by passage, into oov_passages.jsonl, and analyse.py produces every rate, interval and difference above — not every number above, which is a distinction I got wrong in this paragraph before checking it: the counts beside those rates come from the corpus and the draw logs at build time, each of which is published below. The rest of the tree is here too, because a pipeline described in prose is not a pipeline anyone can check: tokens.py measures how much of each passage the 512-token limit cut off, passage by passage, into tokens.jsonl — which is published for the same reason as the rest: analysis.txt prints a truncation rate off it, and the corpus text that would let you recompute that rate yourself is the one thing here that is not published, registers.py holds the single copy of what each query family was looking for and how a passage is attributed to one, clusterci.py computes the intervals that treat books rather than passages as the unit, padding_check.py is the batching check above, selftest.py exercises the build's newest blocks against a partial scan so they do not run for the first time at the moment of publication, and build_page.py renders this page — including every refusal that stops it rendering. The pipeline itself: chain.sh runs the scans and the checks in order, run_scan.sh supervises a scan the kernel may kill and writes its done marker only once the output file has as many lines as the input, finish.sh waits on the chain and produces analysis.txt, and run_analysis.sh is the one copy of the rule that refuses to overwrite a good analysis file with a short one. chain.run.sh and finish.run.sh are the frozen copies that actually executed: a long chain is launched from a copy, because bash reads a script by byte offset and editing the original mid-run corrupts it hours later — and from that moment the source and the running process are two different objects. They are published because they are not identical to their sources, which is to say the sources describe a run that did not happen: the analysis stage exists in chain.sh and not in the copy that executed, so analysis.txt came from the waiter rather than from the chain. Read finish.run.sh and its own header comment will tell you the opposite — that the chain runs the analysis too and this run is a redundant second one. That comment is wrong about this run, and it is published unedited, because a script being read by a live shell cannot be edited without corrupting it, and because the file that ran is the artifact whether or not its author had understood it yet. The corrected account is in finish.sh. One set of scripts is not in this directory at all, and it is the set the selection claim rests on: the passages were cut in the earlier project this grew out of, and merge.py only merges its output, so the check that keeps this paragraph honest — it walks the study directory — was blind to them. fetch_corpus.py is the shared library and the first pull: the archive search, the OCR normaliser, and the passages() cutter both corpora were cut by, which is what “same extraction code” means above. fetch2.py, fetch3.py, fetch4.py, fetch5.py, fetch6.py, fetch7.py, fetch8.py, fetch9.py and fetch10.py are the query families, one per pull, each writing the corpus file of the same number that index.jsonl names for every passage it produced. Anyone who wants to argue that this corpus was selected for something other than machine-looking prose can now read the queries instead of taking the sentence from me. The raw scores are here too: scores_hello.jsonl, scores_openai.jsonl and the control pair, scores_control_hello.jsonl and scores_control_openai.jsonl, one JSON line per passage. Those files key on an integer index, which is unusable on its own, so index.jsonl gives every scored passage its archive.org item, its passage number within that item, its catalogue year, its date-verification tier, its query families and its length — enough to recompute any rate, any breakdown and any confidence interval on this page without rerunning a model, and enough to pull any passage back out of the source scan. What is not published is the adversarial corpus text itself: twelve thousand verbatim runs of OCR from other people's scans. The control corpus is small enough to publish whole, and is, as control.jsonl — along with the discarded first draw, so you can see what was thrown out and satisfy yourself it was thrown out before any control passage had been scored, which is the claim that matters and is not the same as the two files being disjoint. The batching check publishes its per-passage output too — padding_hello.jsonl, padding_openai.jsonl, padding_control_hello.jsonl and padding_control_openai.jsonl give the batched and single-passage score for every passage re-scored, so the worst-case number in that table can be recomputed or contradicted. analysis.txt is the unedited output of analyse.py: every rate, interval and difference on this page appears in it, next to the ones that did not make the page — and the build refuses to render if a number in one of the tables above is missing from that file, which is how I learned that sentence had been false in three places at once. Both corpus files are pinned by digest in corpus.sha256, and the build refuses to render if either one moves; the control file above is published whole, so its digest is yours to check. Because that text is unpublished, index.jsonl carries a sha1 of each passage instead: re-extract the same books and the hashes have to match, one by one. The control draw kept its own record, and that is published too: control.log lists every item the draw considered, the reason each rejected one was rejected — not English, no text, or dated by its own contents rather than by its catalogue entry — and the summary line that the date-rejection count on this page is read out of at build time. One warning for anyone rerunning this: the two checkpoints order their classes in opposite directions — id2label reads {0: Human, 1: ChatGPT} for one and {0: Fake, 1: Real} for the other — so a script that takes the same softmax column for both reports one detector’s human rate as its AI rate. score.py resolves the class by name and exits if the name is absent, and prints the index it resolved to, which is in the run logs.

Run on a one-core box with no GPU. If you find an arithmetic error on this page, the score files above are enough to prove it, which is the point of publishing them.