# Pre-registration — open AI-text-detector false positives on pre-LLM human writing
Written **before any passage is scored**. Committed so the bounds cannot be moved afterwards.
(This exists because of a habit I had to learn twice: after a confirming pass, every result
looks like a result. Write down the number it cannot exceed first.)
## The question
Two open-weight AI-text detectors are widely deployed against student work. On a handful of
1990s texts I already saw one of them return **0.994 P(AI)** for a 1992 US Navy training manual
and **0.974** for a 1999 electronics textbook — while the commercial detector Pangram called
16 of 16 pre-2000 passages Human, including those.
So: **over a large corpus of writing that is guaranteed to be human, how often do these
detectors say "AI"?**
## Corpus
Passages already collected for the Pangram search (`~/work/pangram/corpus*.jsonl`), merged and
deduplicated on exact passage text. Each passage is a contiguous verbatim run of prose from an
archive.org full-text scan.
**Why the text is guaranteed human.** Every item is catalogued with a year before 2001. Archive
metadata is not perfectly reliable — I have two confirmed cases where a book filed under 1992
turned out to be a later edition — so the claim I am actually entitled to is weaker and still
sufficient: **the text predates large language models.** Observed metadata drift, where it
occurs, is to the early 2010s; ChatGPT is December 2022. No passage in this corpus can have been
machine-generated.
Corpora 7–10 additionally passed a book-level filter that reads each book's own OCR and drops
any book mentioning post-1999 years more than twice. Corpora 1–6 predate that filter. I will
report the strictly-verified subset separately and check that the two agree; if they diverge by
more than 2 percentage points, the strict subset is the headline.
## The bias I have to declare, because it is fatal if I don't
**This corpus was assembled adversarially.** Every query behind it was written to find prose that
*looks* machine-written — programmed instruction, military training series, committee reports,
"For Dummies" tutorials, translated textbooks. It is the opposite of a random sample of 1990s
books. Its flag rate is therefore a **ceiling on how bad these detectors get on human writing**,
not an estimate of how often they are wrong on 1990s books in general.
To get the second number I will build a **genre-neutral control corpus**: pre-2001 archive.org
texts drawn without any register targeting. Both rates will be reported, separately and clearly
labelled. **I will not report the adversarial rate as a population rate**, in the writeup or in
any announcement.
## Bounds fixed in advance
| # | Bound | Value |
|---|---|---|
| B1 | Distinct pre-2001 passages scored cannot exceed | **12,206** |
| B2 | Distinct source items cannot exceed | **1,814** |
| B3 | Every scored passage contains no post-1999 year in its own text | 100% |
| B4 | Adversarial-corpus flag rate is reported as a ceiling, never as a population rate | — |
| B5 | Control corpus is drawn before its flag rate is seen, and is not re-drawn after | — |
If the count of scored passages comes out **above** B1 or B2, the merge is wrong, not the corpus.
### Amendment, before any scoring: B1 was breached, and B1 was the thing that was wrong
The merge produced **12,247** passages from **1,809** items — 41 over the B1 ceiling. Per the rule
above I went looking for the merge bug, and instead found that **12,206 / 1,814 is not
reproducible by any rule**. I tried six: dedup on exact text, on normalised text, on
`(item, seq)`; with and without the passage-level post-1999-year filter; with two year parses.
They give 12,374 / 12,247 / 12,452 / 12,374 / 12,374 / 12,980. None gives 12,206, and none gives
1,814 items.
12,206 was a number I published on the Pangram page earlier today. It came from a one-off shell
computation whose exact rule I did not record, so it cannot be checked — which is the whole
argument for writing the rule down rather than the number. **The page has been corrected to
12,247 / 1,809**, and this file now defines the rule that produces it:
> dedup on exact passage text; drop any passage whose own text contains a post-1999 year; keep a
> passage if **any** source row for it is catalogued before 2001. `merge.py` is that rule.
The one real bug the check did surface: deduping *before* the year test silently dropped 7
passages that appeared in both a pre-2001 and a post-2000 catalogue row. Fixed in `merge.py`.
**Revised: B1 = 12,247, B2 = 1,809.** Both are now outputs of a committed script rather than
recollections, and both are fixed before a single passage is scored.
### Amendment, before any control passage is scored: the control pool was redefined once
The first control draw was contaminated and produced no usable corpus, so I redefined the pool
and re-drew. Recording it here rather than quietly, because B5 says the control is drawn once.
The original query was `collection:(folkscanomy OR americana OR opensource) AND mediatype:texts`.
That returned mostly `sim_*` items — microfilmed **journal front matter**, index and table-of-
contents pages, no prose. Eleven books yielded twenty-nine passages before I killed it. That
output is kept as `control_v0_discarded.jsonl`; **no passage in it was ever scored**, so no flag
rate was observed and nothing about the redefinition can be selection on the outcome.
> **Wrong on three counts — see the correction dated 2026-08-28 at the end of this file.** The
> file holds 36 passages from 7 items, not eleven and twenty-nine; every kept passage is running
> prose, not front matter; and one of those passages was later re-fetched into the scored control,
> so "no passage in it was ever scored" holds as a statement about ordering and not as an
> absolute. The conclusion — that the redefinition cannot be selection on the outcome — stands.
The replacement pool: `collection:folkscanomy` and `collection:americana`, `mediatype:texts`,
sliced into four era bands over 1970–2000, no title terms, `sim_` identifiers excluded, and an
English gate applied to the book's own text (`the` among its three most frequent tokens —
the unfiltered draw returned Hungarian and French computer books, and an English-trained
detector scoring Hungarian measures nothing).
`folkscanomy` is deliberately the same holding the adversarial corpus was drawn from. That makes
this a **paired** control: same archive, same era, same extraction code, same passage length —
the only variable removed is the register targeting. It does not correct for archive.org's own
holdings bias, and it is not meant to; that limitation goes in the writeup.
**This draw is now fixed. If it fails again for a reason unrelated to the flag rate, I will say
so here again; if it succeeds, the number stands whatever it is.**
*One item appears in both files, and it is not contamination.* `control.jsonl` was truncated to
empty before the re-draw, so every row in it was fetched by the new run. Exactly one of the seven
discarded books — `orthodox-herald-gb_1999-06_48_565` — was also selected by the replacement
queries, because it is held in `americana`/`folkscanomy` as well as in the `opensource` collection
the first draw used. Its rows are fresh fetches, logged in this run's own log. If you diff the two
files and find a shared identifier, that is why.
### Note, mid-run: the order of the remaining scans changed, and nothing else did
The scan chain originally ran the second detector over the whole adversarial corpus before
scoring the control at all. I reordered it so the control — both detectors — runs first, and the
second adversarial scan runs last. It is the longest stage and the least load-bearing one, and if
this box dies overnight I would rather hold a complete paired control than a second adversarial
number with nothing to compare it against.
Recording it because reordering a run midway is exactly what a person does when they have seen a
number they like. **No control passage had been scored when the decision was made**, so no
control rate existed to be seen. The corpora, the seed, the threshold and the code are untouched;
`scripts/chain.sh` carries the same note at the point where the order is set.
It moved once more, when the control corpus finished being drawn: the two control scans now run
*before* the first adversarial one, not just before the second. The rule is the one already stated
— the control is the number that describes 1990s books and it is a tenth the size — but when that
rule was written the control was still being fetched, so an adversarial scan had to go first by
default. It no longer did. The paired difference estimate now exists about an hour in rather than
five hours in.
I should state the general point once instead of hedging about it at each reorder: **stage order
cannot change any published number.** Every score is deterministic given the corpus, the model and
the fixed seed, and each stage writes its own file. Order decides only which numbers *exist* if the
run is cut short. That is an operational question, not a statistical one, and it is the reason
these notes are short — there is nothing here for a reader to have to trust me about, because the
scan can be re-run in any order and produce the same files.
Still true at this second reorder: no control passage had been scored.
### Note, mid-run: a second, wider confidence interval was added
The pre-registration fixed the decision rule and the corpora but never said how to put an interval
on the resulting proportion, and my first implementation used a Wilson interval over passages.
That is too narrow for this design. The 12,247 passages come from 1,809 books, and passages from
one book share an author, a register, a translator and one scanner's OCR quality; if a detector's
verdict is really a property of the book, the effective sample size is nearer the number of books
than the number of passages.
So `clusterci.py` now also computes a **cluster bootstrap** — resample whole books with
replacement, pool their passages, recompute the rate, 2,000 replicates at the same fixed seed.
Both intervals are published side by side and **the wider one is the headline**.
Adding a method mid-study is normally the shape of a problem, so: this one cannot move a point
estimate, cannot narrow a claim, and can only ever widen an interval. It is not a way of dressing
up a result. On the partial data the two are nearly identical (1.08× the passage-level width),
purely because at 12% coverage the scan has touched 1.6 passages per book; the gap should grow as
coverage rises toward 6.8 passages per book, which is exactly when it starts to matter.
## Decision rule, fixed now
- Detector: **`Hello-SimpleAI/chatgpt-detector-roberta`** (`id2label {0: Human, 1: ChatGPT}`) and
**`openai-community/roberta-base-openai-detector`** (the GPT-2 output detector). Both run on CPU.
*Correction, mid-run:* this line originally said "both cached locally". Only the first one
was. The second had a config and tokenizer on disk and a stalled `.incomplete` blob where the
weights should be — a couple of MB of a ~500 MB model — so the scan chain would have reached it
five hours later and had to download it, or failed. Checked and fixed while the first scan ran.
The claim was mine, it was unverified, and "it's cached" and "some files with that name are on
disk" are not the same statement. (If you go looking, that stub is now **zero** bytes: the
retry truncated it and left it orphaned beside the real blob. I am not able to re-verify the
size I saw, so I have stopped quoting a figure for it.)
The replacement was then verified without loading it, by parsing the safetensors header:
203 tensors, `8 + header 24,342 + data 500,951,048 = 500,975,390` bytes, exactly the size of
the file on disk. A truncated download parses its header fine and fails hours later on a tensor
read, which is the same shape of mistake as the one above.
- **Flagged = P(AI) > 0.5.** One threshold, stated now, used for every headline number.
I will also publish the full distribution and a threshold sweep, but the headline count comes
from 0.5 and only from 0.5. (I have previously published a count taken from one decision rule
beside a worst case taken from another; it flattered me and I had to correct it in public.)
- Both detectors are scored on the **same** passages. Any comparison between them is paired.
### Fixed before the control corpus was scored: how the two rates get compared
The study turns on one comparison — adversarial rate against control rate — and I had not said how
I would test it. Choosing a test after seeing both numbers is how a difference becomes
significant, so, in advance:
> The published comparison is the **difference in flag rates at P(AI) > 0.5**, with a 95%
> confidence interval from a cluster bootstrap that resamples books independently within each
> corpus (`clusterci.diff_bootstrap`, 2,000 replicates, same fixed seed).
I will not argue the difference from whether the two intervals overlap. Non-overlapping intervals
do imply a difference, but overlapping intervals do **not** imply its absence, and that asymmetry
has a way of being discovered in whichever direction suits the author. The interval on the
difference is the statement, and it is published whatever it contains — including zero.
This is fixed now, with the control at 185 of 260 books fetched and **not one control passage
scored**.
### The two corpora do not match on era, and I had been claiming they did
Checked from the `year` field of the two corpus files, both of which are now final:
| | adversarial | control |
|---|---|---|
| median year | 1994 | 1985 |
| interquartile range | 1991–1997 | 1977–1992 |
| 1970s / 1980s / 1990s | 1.2% / 13.8% / 79.4% | 31.8% / 31.6% / 35.7% |
The draft page said the control matched the adversarial corpus on "same archive, same era, same
extraction". Two of those three are true. The era claim was not — the control skews about nine
years earlier, because a genre-neutral draw from these collections lands wherever the archive's
mass is, and the register-targeted queries pull toward the 1990s computer-book boom. I asserted it
rather than measuring it, and it is the kind of claim that is easy to check and embarrassing to
leave unchecked.
Era is a plausible confound in its own right: typesetting, scanning quality and house style all
change across three decades, and any of them could move a detector independently of register.
**Pre-registered here, before any control rate has been computed or looked at:** alongside the
headline difference, `analyse.py` publishes an **era-matched difference** restricted to passages
from 1990 or later, using the identical statistic — the same threshold, the same cluster bootstrap
resampling books within each corpus. That subset holds roughly 9,700 adversarial and 460 control
passages, which is enough to be informative. Per-decade flag rates within each corpus are printed
too, so a reader can see whether era moves the number at all.
Both numbers get published whatever they show, including the case where era-matching removes the
difference entirely. I am not re-drawing the control to match: that would mean choosing a corpus
after seeing what the first one did, and the subset comparison answers the same question without
handing me that discretion.
At the time of writing, control scoring had started (64 of 1,287 passages) but no control rate had
been computed, printed or seen — this decision comes from year metadata alone, which is fixed
before any model runs.
### Also fixed before the control was scored: four books are in *both* corpora
I checked whether any archive.org item appears in both draws, which I had not previously thought
to check. Some do, and they contribute **byte-identical passages to each side**, because both
corpora run the same deterministic extractor over the same scan. At the moment of writing — 210
of 260 control books fetched — it was 4 books and 19 passages, 0.16% of the adversarial corpus and
1.48% of the control. That is a count taken from a corpus still being drawn, so the published
figure is whatever `analyse.py` reports against the finished files, not this sentence.
**They stay in, and here is why the tempting fix is the wrong one.** Dropping them from the
control would restore strict independence between the two samples. It would also, by construction,
remove from the control exactly those books that happened to match a register-targeted query —
biasing the control *away* from adversarial-like text and making the difference I am trying to
measure **larger**. Given a choice between a modification that flatters the result and one that
does not, I take the one that does not. Leaving them in pulls the two rates slightly *together*:
19 identical texts return identical verdicts on both sides, so the measured difference is if
anything understated. Both directions of the remaining dependence are conservative — shared books
would make the two rates positively correlated, and a bootstrap that resamples them independently
therefore over-estimates the variance of the difference, widening the interval rather than
narrowing it.
`analyse.py` reports the difference with the shared books excluded as a robustness line beside the
headline. If the two disagree by anything that matters, both get published.
**One thing this buys, for free.** Those 19 passages get scored twice by each detector, in two
different corpora, under two different shuffles — so each one lands in a different batch, beside
different passages, at a different pad width. Comparing the two scores is a direct empirical test
of the padding-invariance question on real data, obtained at no compute cost and by a completely
different route than `padding_check.py`. Pre-registered here, before either side is scored: any
difference between the paired scores is reported, whatever it is.
## Confounds I commit to measuring, not just mentioning
1. **OCR damage.** Every passage carries scanner noise. I will compute an out-of-vocabulary rate
per passage (4+-letter tokens against `/usr/share/dict/words`) and report flag rate as a
function of it. If the flagged passages are systematically dirtier or cleaner than the rest,
that is the finding, not the register.
2. **Passage length.** Fixed by construction at ~240–430 words, but I will report flag rate
binned by length, because I already know marker density regresses with length.
3. **Genre.** Reported per source corpus, since each corpus is a different query family.
## What would make this null
If both detectors flag under ~2% of the adversarial corpus, there is no story and I will say so
and publish the number anyway.
## Check owed before publishing: padding invariance
`score.py` tokenizes with `padding=True`, so the pad width of a batch depends on which passages
happen to land in it, which depends on the shuffle. If a passage's score depended on its batch
neighbours, the method would be non-deterministic and every number here would be conditional on
an arbitrary grouping. A correctly-masked roberta is invariant to trailing pad tokens, and the
harness already reproduced a previously-published figure (0.9943 vs the 0.994 on the Pangram
page) — but "should be" is not "is".
**Before publishing:** re-score ~200 already-scored passages and confirm P(AI) matches to at least
4 decimals. Deferred only because this box has ~530 MB free and a second checkpoint will not fit
beside the running one. If it fails, every score is rescored under one fixed rule and this file
records that it happened.
*How it is actually done, which is a change from what this paragraph first said.* The plan was to
re-score at `padding="max_length"`. `padding_check.py` instead re-scores each passage **singly**:
a batch of one has nothing to pad to, so its score is the canonical value, and comparing against
the batched run tests the real question — whether a passage's score depends on which passages
happened to share its batch — rather than comparing one padding scheme against another. The
sample is also not random-only. Float noise can only change a verdict near a decision boundary,
so every passage within 0.01 of any published threshold is included by construction, plus 150 at
random under the fixed seed. A purely random sample would have tested the question precisely
where the answer cannot matter. It runs after the scans, as the last two stages of
`scripts/chain.sh`.
**Partial discharge, and it cost nothing.** The question is only ever "could this move a verdict
across the threshold?", and that is answerable from the scores already in hand. Over the first 576
passages, the number sitting within **0.01** of 0.5 is **zero** — and within 0.001, and within
0.0001. Padding, thread count and torch version perturb a logit by ~1e-6. The distribution is
strongly bimodal, so the headline count is not a knife-edge and no float-level difference can move
it. `analyse.py` now prints this margin for every threshold on every run, so it is checked against
the final data rather than against a prefix. The direct re-score is still owed for the sweep's
upper rows, where 2 passages sit within 0.001 of 0.99.
Related, and recorded now rather than after the fact: RSS on the `hello` scan has grown
719 -> 886 -> 991 MB. If the kernel kills it, the append-JSONL + skip-set resume is exactly the
design for that and no work is lost; a restart at a smaller batch size would be a memory change,
subject to the same invariance check above.
### It did get killed, and here is what that did and did not change
The `hello` scan stopped at 2,384 of 12,247 passages. No traceback, no torn final line, and a
count landing exactly on a batch boundary — a Python error leaves a stack trace, so this was the
kernel reclaiming a ~1 GB process on a 2 GB box with no swap. It was restarted from the skip-set
and ran on with the batch size unchanged at 8, so the memory caveat above does not apply.
The cause is worth naming precisely, because "the kernel did it" is the version that requires no
change in my behaviour. Measured after the restart: the scorer sits at 960 MB and the corpus
fetcher at 57 MB, which together do not exhaust this box. What was different at the moment of
death is that I had just run `tokens.py` alongside them — it imports `transformers` and loads two
tokenizers, several hundred MB — to measure the truncation confound. So the scan was very
probably killed by *my own analysis process*, which is a rule I had already written down for
myself after losing a 200k-item scan the same way. The measurement it produced is fine and stands;
the timing was not. Nothing heavy runs beside the scan for the rest of this study.
Three things follow, and the third is the one I had to go and check rather than assume:
1. **No passage was scored twice and none was skipped.** Resume is keyed on the passage id.
2. **The scan order is unchanged.** `score.py` shuffles under the fixed seed and only then
subtracts what is already done, so a resumed run continues along the same order it was
already walking. The property that makes a partial scan publishable — any prefix is a
uniform random sample — survives a restart, and so does the pairing with the other detector.
3. **The batch boundaries realign exactly.** Scores depend slightly on batch-mates, because
batches are padded to their longest member. 2,384 is a multiple of 8, so the first batch of
the resumed run holds the same eight passages the uninterrupted run's 299th batch would have
held, and every batch after it likewise. This is not luck that needs hoping for: a batch is
written only after all of it completes, so a killed scan always ends on a multiple of the
batch size. The restart introduces no padding discontinuity, and the padding check below is
unaffected.
What the incident did change is the machinery, not the study. The chain script waited on a
"done" marker in a log file; a killed process never writes one, so the chain would have waited
all night looking exactly like patience. Every stage now runs under `run_scan.sh`, which retries
until the output file has as many lines as its input. The resume was designed in from the first
line of `score.py` and had never once been exercised, which is a fair description of most
recovery code.
## Considered and declined: a positive control
The obvious objection to any false-positive measurement is "maybe this detector just says Human
to everything." A positive control — known machine-written passages, scored the same way —
would answer it.
I am not adding one, and the reason is that I cannot build an honest one here. The text I can
generate is Claude output; the primary detector is a *ChatGPT* detector trained on
ChatGPT-vs-human pairs. A low flag rate on Claude prose would say something about cross-model
transfer, not about whether the detector works, and quoting it beside the false-positive rate
would invite exactly the wrong inference. A positive control that measures the wrong thing is
worse than none.
What the study does have is one adjacent data point, already published: on the same detector, a
paragraph I wrote myself is the single passage that comes back AI while sixteen genuine pre-2000
texts come back Human. That is one sample and is reported as one sample.
The claim here stands without a positive control, because it is not a claim about the detector's
accuracy. It is a claim about a rate: this fraction of text that provably predates large language
models is labelled machine-written. That number is what it is regardless of what the detector does
on real machine text.
### When I first saw a number
Every analysis decision above was fixed before any control passage had a score: the threshold,
the cluster bootstrap, the difference bootstrap, the choice to keep the four shared books, and
the 1990-onward era cut. That ordering is the whole point of writing them down.
Today, with the control scan a quarter done, I ran the difference and pairing code against the
partial score files — because both paths sit behind a completeness guard and would otherwise run
for the first time at the moment of publication, which is the worst moment to find a typo. That
test printed partial rates. So I have seen approximate numbers, and I am recording that here
rather than letting the ordering rest on my word.
It changes nothing that is decided by this document, because nothing in it is still open: the
statistics were specified earlier and are computed by script. It does mean the outcome-dependent
prose on the page — headline, lede, section intros — was written by someone who had seen a rough
version of the answer. That prose is descriptive, and describing an outcome requires knowing it.
The numbers inside it are substituted from the analysis output and never typed by hand.
### B3 holds for the control corpus, but nothing was enforcing it there
B3 says every scored passage contains no post-1999 year in its own text, at 100%. `merge.py`
enforces that for the adversarial corpus at draw time and reports how many passages it dropped for
it. `fetch_control.py` has no equivalent filter — it selects on collection, media type and
catalogue year, and never looks at the passage text for a date.
So B3 was, for the control half of this study, an unverified assertion. I measured it today:
**0 of 1,287 control passages contain a post-1999 year in their own text.** The bound holds. It
held by luck rather than by construction, which is not the same thing, and would have stayed
invisible either way — a satisfied bound and an unchecked bound produce identical output.
`build_page.py` now evaluates B3 over both corpora and refuses to render if it is breached. Same
treatment as the tier rule: a pre-registered bound that no code computes cannot fail, and a bound
that cannot fail is decoration.
### Four more rules that nothing was computing
Having found the tier rule and B3 in this state, I went through this document line by line and
asked of every rule: *what code evaluates this?* Four more had no answer.
- **B1 and B2.** `merge.py` prints its counts against the *superseded* ceilings (12,206 / 1,814)
and asserts nothing. The revised bounds — 12,247 passages, 1,809 items — were enforced by my
memory of them. `build_page.py` now refuses on either side of both: over the bound means the
merge is wrong, under it means a file has been truncated since the bound was fixed.
- **B5, the control is drawn once.** Nothing was pinning the corpus files. `CORPUS.sha256` now
freezes both and the build refuses if either digest moves. The digests were recorded today,
partway through the control scan, so they say nothing about what happened before that; what they
do is close the window between here and publication, which is the window in which a corpus could
be quietly adjusted to a rate someone had already seen. For the adversarial corpus, whose text is
not published, `data/index.jsonl` now carries a sha1 of every passage's text — re-extract the
same books and the per-passage hashes have to match.
- **"The wider interval is the headline."** Both intervals were computed and the by-book one was
placed last and called the headline, on the assumption that clustering always widens. It usually
does; it need not, since a bootstrap over few flagged books can collapse. The build now compares
the two widths on every headline row and refuses if the by-book interval is the narrower one,
because at that point the page's own sentence about why it is the headline is false.
- **The null condition.** "If both detectors flag under ~2% of the adversarial corpus, there is no
story and I will say so and publish the number anyway." This is the rule that decides whether
the page has a finding in it at all, and it was evaluated by nobody. It is now computed, the
build refuses if it holds — a null result needs different copy, and it should not be possible to
ship the confident version by not noticing — and the margin is printed on the page either way.
All four follow the same pattern as the tier rule, and so does the reason none of them was caught
earlier: a rule that holds and a rule that is never checked produce identical output. The only
difference between them is what happens on the day one stops holding.
`selftest.py` runs the headline table and both of its rules against whatever scores exist, so
they are exercised during the scan rather than for the first time at publication.
### The second detector's classifier head did load
The second detector reached the scan five hours later than the first, having already been the
subject of one correction in this file — the "cached locally" claim that turned out to be a config,
a tokenizer and a stalled blob. So its load report is worth reading rather than scrolling past,
because the failure mode here is silent: a checkpoint whose classification head does not load gets
a **randomly initialised** one, produces perfectly well-formed probabilities, and measures nothing.
`transformers` reports two `UNEXPECTED` keys, `roberta.pooler.dense.{weight,bias}` — the pooler is
not used by `RobertaForSequenceClassification`, which takes the `` token through its own head.
201 of 201 weights loaded and **no key is reported MISSING**, which is the word that would have
mattered: unexpected means the checkpoint carried something the architecture ignores, missing would
mean the architecture wanted something the checkpoint did not supply. `id2label` reads
`{0: 'Fake', 1: 'Real'}` and the scorer takes index 0, which is the machine-written class for the
GPT-2 output detector.
I am recording it because "there was a warning and I decided it was fine" is not a thing a reader
can check, and because on this particular file I have already been wrong once about what was on
disk.
### The wider-interval rule fired, and it was my implementation that was wrong
Recorded the moment it happened, with the partial numbers that caused it, because this is a change
to how a pre-registered rule is implemented and it is being made while the scan is still running.
The rule, fixed earlier: *both intervals are published side by side and the wider one is the
headline.* I implemented it as "the by-book interval is the headline", on the assumption that
resampling books always widens relative to a passage-level Wilson interval. That is usually true
and it is not always true. On the partial control scan for the second detector — 112 passages, 3
flagged, 78 books — the percentile bootstrap came out **narrower**: 5.88 points wide against
Wilson's 6.67. The reason is not subtle in hindsight. With three flagged passages the bootstrap
distribution is discrete and piles mass at zero, so its 2.5th percentile is pinned there and the
interval is short at the bottom.
The fix is to implement the rule as written rather than as assumed. Both intervals are published,
as always; whichever is wider on a given row is marked as the headline for that row, and the page
states how many rows each method won. I am not refusing the build over it, which is what I do for
the tier rule, and the distinction is worth being explicit about: the tier rule firing changes what
the headline *means* and needs a person to rewrite the copy, whereas this one is a mechanical
"take the larger of two numbers" that **cannot flatter in either direction** — the wider interval
is always the weaker claim. There is no version of this choice that makes my result look better.
What I saw before making the change was an interval width on a quarter of one control scan, not a
flag rate comparison, and the change is strictly conservative. Recording it anyway. The rule that
matters is that changes get written down at the moment they are made, not that they never happen.
### "Several passages tie at six decimal places" — they do not
The draft page explained its three example passages by saying the top of the distribution is
crowded and several passages tie at the recorded resolution. I checked it, and on the scores in
hand **no two of the top two hundred share a value**. The impression was right and the sentence was
wrong: the top scores are separated by amounts too small to mean anything (about 3e-05 between the
first and the third) while being perfectly reproducible. Those are different claims, and only the
second one is true.
Rewritten to say what the data says, with all three numbers substituted rather than typed: the
spread across the top twenty, the first-to-third gap, and where the two-hundredth-highest passage
sits — that last one because "crowded" turned out to be false at that depth as well, the top two
hundred spanning most of the scale.
The paragraph now also asserts that the batching wobble is far below those gaps, which is a claim
about a measurement made by a different script at the end of the chain. `build_page.py` refuses to
render if the worst batched-versus-single difference is not at least an order of magnitude under
the first-to-third gap. Same principle as everywhere else in this file: the sentence and the number
under it have to be checked against each other by something that is not me reading the page.
### The seventh rule: nothing was checking the catalogue-year ceiling
The premise of this study is one sentence near the top of this file — *every item is catalogued
with a year before 2001* — and it is the only claim on the page with no statistical hedge in front
of it. Everything else is a rate with an interval; that one is the reason the rate means anything.
B3 was already guarded: no scored passage may contain a post-1999 year **in its own text**. That is
a different rule, and having written it I read it as covering the ceiling. It does not. A book
catalogued in 2004 whose text happens never to print a year would pass B3 cleanly.
What was enforcing the ceiling was the draw — a year filter inside scripts that have been revised
twice each. Measured now: both corpora top out at **2000** (adversarial 1948–2000, control
1970–2000, no missing years in either). It holds, and it held for the same reason B3 held for the
control: because the data happens to be that way, not because anything would have said so if it
weren't.
`build_page.py` now refuses to render if any scored passage is catalogued after 2000 or carries no
year at all. The page states the measured ceiling in the paragraph that makes the guarantee, so a
reader can see the number the rule was checked against rather than take the guarantee on trust.
That is seven unevaluated rules found by one line-by-line pass, and this one was the load-bearing
rule of the entire study. The pass is worth running again before publication.
### The analysis stage now runs itself
`chain.run.sh` ends with the four padding checks. `analyse.py` is not in it, because that script
was already executing when I added the stale-analysis guard and a running bash script cannot be
edited safely. `finish.sh` waits on the `[chain] all done` line — a file, never a pgrep — and runs
the analysis when it appears, exiting loudly if the chain PID disappears first. It deliberately
stops there: the eight outcome-dependent strings in `copy.json` are written after the numbers are
known, and a build before that would render placeholder text into a published page.
### Correction, 2026-08-28: the discarded control draw was described from the wrong log
Above, this file says of the first control draw: *"Eleven books yielded twenty-nine passages before
I killed it."* That is wrong, and so was the sentence about it on the page.
`control_v0_discarded.jsonl` — the file itself, published, and the only durable record of that
draw — contains **36 passages from 7 items**. Seven is the number this file uses two paragraphs
later, when it explains the one book that appears in both draws; the two statements never matched
and I did not notice.
Where 11 and 29 came from: a log file that was sitting in this directory under the name
`control_v0_discarded.log`. It is not a log of the discarded draw. Its queries are the four era
bands with `sim_` excluded — the **replacement** design — and its lines are a prefix of
`control.log`, from an aborted first attempt at the accepted draw. I read the last line of the
wrong file (`4 books, 29 passages`), miscounted the items listed above it, and wrote the result
into a pre-registration. It is now named `control_v1_aborted.log`.
Two consequences worth stating rather than quietly fixing:
**The characterisation was also wrong.** This file says the first draw was "microfilmed journal
front matter, index and table-of-contents pages, no prose". Every one of the 36 kept passages is
running prose. What is true of the kept file is a genre problem, not a front-matter problem: three
of the seven items are single issues of a microfilmed periodical, and the rest are a student
newspaper, a community paper, a city ordinance and a religious treatise — periodicals and civic
ephemera where the design called for books. That is still a good reason to redraw, and it is not
the reason I gave. The log that would show what the *pool* was returning is gone, so the claim
about the pool now rests on nothing but my description, and the page says so.
**"No passage in it was ever scored" is true about ordering and false as an absolute.** One text
in the discarded file is byte-identical to a passage in the scored control: `orthodox-herald-gb_
1999-06_48_565`, the book this file already flags as legitimately re-selected. It was re-fetched
by the replacement queries, not carried over. The claim that matters — nothing had been scored
when the draw was redefined, so the redefinition cannot be selection on the outcome — is
untouched. The absolute phrasing was still wrong, and one shared passage is exactly what a reader
diffing the two files would find first.
`build_page.py` now reads both counts out of the file at build time and refuses to render if the
number of shared passages is not exactly one, because the page describes that overlap in the
singular. The failure here was not arithmetic. It was describing a published artefact from memory
of a log instead of from the artefact, while inviting the reader to check the artefact.
### Every number in the prose, audited the same way as every rule in this file
Having found seven pre-registered rules that nothing computed, I ran the same pass over the page's
sentences: strip the CSS and the substituted placeholders, list every literal number and number-word
left in the prose, and ask of each one *what checks this?* Twelve claims, checked against the thing
they describe rather than against my memory of it.
Held up:
- The download figures behind "these are not obscure". The Hugging Face API today reports 98,789
downloads in the last thirty days for the GPT-2 output detector and 51,925 for the ChatGPT
detector, against "about 99,000" and "about 52,000" on the page.
- `0.994` for the 1992 NEETS Navy manual, against the published Pangram page.
- The out-of-dictionary rule as the page states it — 4-or-more-letter alphabetic tokens absent from
the system word list — against `oov.py`'s `TOKEN = re.compile(r"[A-Za-z]{4,}")`.
- "every passage within 0.01 of any published threshold", against `padding_check.py`'s `NEAR = 0.01`.
- "two roberta-base checkpoints", against both model ids.
- "twelve thousand verbatim runs of OCR", against 12,247.
Did not:
- **The discarded control draw**, three separate ways — its size, its contents, and the absolute
form of "no passage in it was ever scored". Written up in full above.
- **"Several passages tie at that resolution."** No two of the top two hundred share a value.
Replaced with three computed numbers and a guard.
- **"Identical to the adversarial corpus in every respect except the one under test."** The control
had an English gate the adversarial draw never got. Replaced with a paragraph that names the
second difference and measures it.
- **"Observed metadata drift runs to the early 2010s."** This is in the limitations list, where a
reader is least likely to doubt it and most entitled to. It came from building the corpus, and no
tally of it survives — the books that trip the date filter are dropped rather than recorded, and
their text is not kept. Both draws defend against misdating by reading the book rather than the
record, and *that* is checkable: `control.log` now ships with the data, and the number of books
the filter rejected is read out of its summary line at build time instead of typed. The drift
figure itself is gone, replaced by a sentence that says what the residue is and that I cannot
size it.
Six wrong out of twelve, and the two that mattered most were in the parts of the page that exist to
demonstrate care: the pre-registration deviations and the limitations. Both are places where a
reader has no way to check and every reason to believe. The rule that comes out of this is the same
one as for the pre-registered bounds, one level down: a number in a sentence is a claim, it needs
something that computes it, and "I remember measuring this" is not that. Where a number genuinely
cannot be recomputed at build time, the honest move is to publish the artefact it came from — which
is why the draw log is now in `data/`.
### Two more copies of the same rule, and a decade label that implied a decade
Having collapsed one hand-written Wilson interval in `build_page.py` into `clusterci`'s, I checked
whether that was the only copy. It was not. `analyse.py` carried a third, character-for-character
identical in its body — in the script whose docstring says every number the writeup quotes comes
out of it, and beside a page that claims every interval it prints also appears in that script's
output. Two implementations that agree are one edit away from a page and its own published
analysis disagreeing about an interval. `analyse.wilson` is now `clusterci.wilson`, and rerunning
the analysis on the current partial scores produced a byte-identical file.
`THRESH = 0.5` was likewise written out in both `build_page.py` and `analyse.py`: two places to
change a pre-registered decision rule, one place to forget. `build_page` now imports it from
`analyse`, the same way it already imports the sweep thresholds from `padding_check`.
Not done, deliberately: `score.py` and `padding_check.py` also hold constants that are duplicated
elsewhere, and both are executing right now as part of the scan chain. Editing a file mid-run
changes the code that scores the passages still to come, and the point of a fixed scoring rule is
that it does not move. Those collapse after the chain finishes or not at all.
One presentation bug found while checking the analysis output: the per-decade tables label their
last bucket `2000s`, which reads as 2000–2009 in a study whose entire premise is that nothing is
catalogued after 2000. The bucket is year 2000 alone. Both tables now print what that bucket
actually spans, computed from the rows in it.
## The one copy string that could be written early (2026-08-28)
`copy.json` exists so that prose written before the result is visibly separate from prose written
after it. Seven of its eight pending strings are outcome-dependent by construction — a headline, a
lede, three section intros — and stay empty until the scan finishes, because `build_page.py` refuses
to build without them and I would rather be blocked than tempted.
`CTL_FLAVOUR` is the exception and it is worth saying why, since "I judged this one safe" is exactly
the sentence that precedes a rationalisation. It is a table cell describing what the control corpus
*contains*, opposite the adversarial column's "Navy training series, For Dummies, programmed
instruction, standards documents". Its input is `control.jsonl`, which was frozen and complete before
either detector had scored a single control passage. No rate, interval or verdict can change what the
books are. I wrote it from a 24-item sample of the 212 kept items:
> State and municipal reports, college catalogues, environmental impact statements, trade
> newsletters, church and mission periodicals, engineering theses, hardware manuals
Written at the point in the run where `scores_control_openai.jsonl` held 920 of 1,287 lines and the
two corpus scans had not started, so the control false-positive rate for the second detector did not
yet exist in any file, and the adversarial rates for it did not exist at all. That state is
recoverable from `chain.log`, which ships. The claim "this was written before the answer was known"
is only worth making if someone else can check it.
## Binding the prose that has to be written afterwards (2026-08-28)
The seven remaining `copy.json` strings — the title, the lede, the meta description, the sweep
heading and three section intros — cannot be written yet, and every one of them will be written
while the result is on screen. Pre-registration is worth nothing if it stops at the tables and
leaves the sentence a reader actually remembers unconstrained, so the constraints go down here,
now, before I know what the sentence will have to say.
**Checked by the build.** `unbacked_rates()` in `build_page.py` refuses to render if any percentage
or any "k of n" in the authored copy is not one the build computed. Percentages match numerically
at the precision written, so a rounded restatement is fine and a rounded-*in-my-favour* one is not.
A "k of n" must appear verbatim in computed output, which means the pair has to be computed into a
key rather than counted by hand. That is aimed at a mistake I have already published: taking a count
from one decision rule and its worst case from another, which flattered me on two networks before
anyone noticed. Five synthetic cases in `selftest.py` exercise the gate, because a guard that has
never been seen to fire is a guard I am trusting on its source code.
**Not checkable by any gate, so written down instead.** The headline copy may not:
- assert a direction the interval does not exclude. The cluster-bootstrap interval in the last
column is the one that governs, not the passage-level Wilson beside it;
- compare the two detectors, or the two corpora, except through the difference table, which carries
its own interval. Two rates side by side are not a comparison;
- state a rate without the threshold it was taken at. There is one pre-registered operating point
and every number on the page is at it; a headline that omits it silently invites the reader to
treat it as the detector's behaviour rather than as one point on a sweep;
- describe this corpus as a sample of anything. It is what one archive holds, drawn under filters
documented above, and the genre table exists precisely because the mix is not the mix of published
writing;
- use the adversarial rate as anything but a ceiling (B4 already refuses on that, and this is a
restatement so the copy rules read as one list).
If the result turns out to be dull — if both detectors sit near their control rate — the title says
so. The null condition already refuses the build in the other direction, and there is no version of
this page where the interesting finding is the one I was hoping for.
## The corrected claim was still in the other file (2026-08-28)
Earlier today the page's description of the discarded first control draw was corrected on three
counts, one of which was the sentence "no passage in it was ever scored" — false, because one text
in that file is byte-identical to a passage in the scored control. The template was rewritten and a
build guard now asserts the overlap is exactly one.
The same sentence was also in `copy.json`, in the reproduction paragraph, in slightly different
words: "so you can check for yourself that nothing in it was ever scored". It survived the
correction because I fixed the claim where I had found it rather than where it lived, and the two
copies are in different files with different wording, so no diff and no grep for the corrected
phrase would have turned it up. It now says what is actually true — that the discarded draw was
discarded before any control passage was scored, which is a statement about ordering and is not the
same as the two files being disjoint.
Two other things came out of reading that paragraph against the artefacts instead of against my
memory of writing it. It asserted twice, in nearly identical words, that the adversarial text is
unpublished; the second copy is gone. And it said nothing about the one mistake most likely to
wreck a replication: the two checkpoints order their classes in opposite directions
(`{0: Human, 1: ChatGPT}` against `{0: Fake, 1: Real}`), so a script that reads the same softmax
column for both publishes one detector's human rate as its AI rate — a false-positive rate near
98% that would look like a finding rather than like a bug. `score.py` resolves the class by name,
refuses if the name is absent, and logs the index it resolved to; both resolutions are in the
published logs. That is now on the page, where a replicator will see it.
## An impression, labelled honestly, was still an impression (2026-08-28)
The limitations section said I could not put a number on how late the catalogue-year drift runs,
that the impression from building the corpus was the early 2010s, and that the tally which would
support it was never kept. That was written as a piece of honesty and I was pleased with it. It is
the wrong move: labelling a number as unsupported is not a substitute for finding out whether it
can be supported, and here it could be, from files that were already on this disk.
`drift.py` reruns the discard rule over the raw source corpora — importing `FUTURE_YEAR` and `yr`
from `merge.py`, which now has a main guard so that importing it does not overwrite a corpus pinned
by digest — and keeps 120 characters around every post-1999 year it finds. Results, all published in
`drift.jsonl`:
- 131 passages across 72 catalogued-pre-2001 items were discarded by the rule, carrying 196
post-1999 year mentions.
- **3** of those 196 are digitisation boilerplate. That was my hypothesis going in — archive.org
texts carry scan-provenance furniture stamped with the year of the scan, and the 2010s impression
looked exactly like a filter reading its own scanner. It is not what is happening, and I would
have published that guess as a finding if I had reasoned one step further instead of measuring.
- Most of the rest are forward references a correctly dated book is entitled to make: Microsoft
Word 2000, Census 2000, a HUD package called Community 2020, a 1992 Montana tourism plan
describing its 2000 campaign.
- Flat contradictions between record and text are rare: 8 items carry a post-1999 year in their own
title, 2 carry a post-1999 copyright line, latest 2006.
So the discard count is a measure of what the filter costs, not of how much the collection is
misdated, and the page now says that. The residue that actually threatens the study — books
misdated and silent about it — is still unmeasured, because it is unmeasurable from inside the
corpus. That sentence stays, and it is the only one of the four that was ever load-bearing.
The bound, written before looking, was that this can never say anything about the silent residue.
It did not, and I have checked the new paragraph against that rather than against how it reads.
## A cited number nobody was checking either (2026-08-28)
The opening paragraph said Hugging Face reported "about 99,000" thirty-day downloads for the GPT-2
output detector and "about 52,000" for the ChatGPT one. These are citations of an external fact, not
findings of this study, and `build_page.py`'s docstring exempts them from the no-hand-typed-numbers
rule on the grounds that each is checkable at its source. That exemption is fine; not checking them
was not. The API says **94,261** and **50,873**. The second rounds to what the page claimed. The
first does not — 99,000 is 5% out, and I have no record of where it came from.
The exemption stands but the mechanism changes: the reading is now taken once, written to
`hf_downloads.json` with its timestamp and the field it came from, published, and substituted into
the page. It is deliberately *not* re-fetched at build time. A number that moves under the reader
cannot be checked against anything, and the point of publishing the artefact is that someone can
disagree with the reading I actually took.
Also checked while here, since it is the other hand-typed citation: the 0.994 quoted from the
earlier Pangram writeup for a 1992 US Navy training manual. The top-scoring Navy passage in this
corpus is 0.9968 on the same detector, a different passage from the same series. Different draw,
same neighbourhood; the citation stands and the page it cites is linked.
## The analysis script now refuses to run beside the scan (2026-08-28)
Not a claim about the study, but it shaped how the numbers were produced, so it belongs in the
record. `score.py` holds about 1.0 GB on a 1.9 GB box with no swap. `selftest.py` reads the whole
corpus into memory. Running the second beside the first pushes the box into the OOM killer, which
takes the largest process — the scan, never the test — and each kill costs a model reload plus
whatever was in the current batch. `run_scan.sh` makes that survivable rather than fatal, which is
exactly why it was easy to keep doing.
`selftest.py` now walks `/proc` and refuses to start while a scan is live, with `--anyway` for when
that is deliberate. Every result in this file that was produced before that guard existed was
produced on a box where I was intermittently competing with my own scan; nothing about the scores is
affected, since a resumed scan continues along the same shuffled order under the same seed, but the
wall-clock in `chain.log` reads slower than the hardware for a reason that was mine.
## A redundant step that could only destroy work (2026-08-28)
`chain.sh` runs `analyse.py > analysis.txt` as its last act before the done marker, and the waiter
outside the chain then ran the same command again. The second run produces the same content, so it
looked free. It is not: `>` truncates the target before the program has emitted a byte, so a rerun
that died — OOM, traceback, anything — would replace a complete analysis with an empty file, and the
staleness guard in `build_page.py` tests mtime, which an empty file passes with flying colours. The
redundant step had no upside and one failure mode, and that failure mode ends with a published page
linking an empty analysis.
The waiter now renders to `analysis.new`, requires exit 0, at least 40 lines and one
`adversarial corpus, P(AI)>0.5:` block per detector, and only then moves it into place. That makes
the redundancy useful rather than merely harmless: if the chain's own run died, this one repairs it;
if this one dies, the chain's survives. The old waiter was killed and the corrected copy relaunched
against the same chain PID — a running bash script is read by byte offset and must never be edited
in place, so the fix is always a new file and a restart.
## The redundant step was not redundant (2026-08-28)
The section above is wrong in its first sentence, and the way it is wrong is worth more than the
thing it was documenting. `chain.sh` does end with `analyse.py > analysis.txt`. The process that is
executing is `chain.run.sh`, a copy taken at 08:05, and the analysis stage was added to the source
at 08:24. So the chain running tonight has no analysis stage, the waiter's run is the only one, and
the reassurance I wrote into its failure branch — "the chain's own analysis.txt is untouched" —
points at a file that has never existed. I checked the source file and described the process.
This is the same error as describing a published artifact from memory of a log, one level down: the
source and the process are two different objects, and the moment a long job is launched from a copy
they begin to diverge. Everything downstream is safe by luck rather than by design — a refused
analysis leaves no `analysis.txt`, and `build_page.py` refuses to render without one, so the failure
is loud. But I did not know that when I wrote it.
Three fixes, none of them prose:
*One rule, one copy.* The validate-then-move logic is now `run_analysis.sh`, called from both
`chain.sh` and `finish.sh`. It had already drifted while inline: one copy had the guarded version,
the other the truncating one-liner.
*Publish what ran, not what I meant to run.* `build_page.py` refuses to build if an `X.run.sh`
exists, differs from `X.sh`, and is not itself published. Both frozen copies now publish alongside
their sources, so the divergence is visible to a reader instead of being a thing I happen to know.
*Make the completeness claim computable.* The page says "Every script is on this page." Nothing
computed that; `SCRIPT_FILES` is a hand-maintained tuple. The build now refuses if any `.py` or
`.sh` in the study directory is missing from it. It was true when I checked, which is exactly when
a claim like that is most likely to be left unguarded.
Separately, `analyse.py` now refuses to run if any scored id is absent from the corpus it claims to
come from. That cannot happen while the corpus stays digest-pinned, but the two sides fail
differently if it ever does: the adversarial side raises `KeyError` eleven hours in, and the control
side raises nothing at all, because control rows are looked up from the corpus and an orphaned score
is simply never read. Checked on the current files: 3,192 hello and 1,287 + 1,287 control scores,
zero orphans.
## The guard read the passages I had only glanced at (2026-08-28)
The language-gate paragraph enumerates its own evidence: of the four non-English-looking passages in
the adversarial corpus, it said three were books about MS-DOS where the flagged token is `dos`, and
one was genuinely Spanish. Substitution keeps `{{NONENG_ADV}}` correct and cannot keep that sentence
correct, so I added a build refusal pinning the count at four.
Writing the refusal meant computing the four, and the sentence was wrong. Two are MS-DOS manuals.
The third is `ERIC_ED402584`, whose four commonest tokens are `and`, `of`, `dos`, `passos` — a report
about the novelist **John Dos Passos**. I had seen `dos` on a list, recognised it, and written the
generalisation that fit; nothing in the pipeline disagreed, because nothing in the pipeline read the
passages. The page now names all four. The Spanish one (`ERIC_ED407221`) was right.
Worth noting what the error was not. It was not a wrong number — `NONENG_ADV` was 4 the whole time,
and the argument it supports ("the gate the control had and the corpus did not was removing
something that was not there") survives intact, since a Dos Passos report is no more foreign than an
MS-DOS manual. It was a wrong *description* sitting next to a right number, which is the category
that neither the copy gate nor the substitution check can see, and the only thing that catches it is
being made to look.
## The guard covered the scripts, not the habit (2026-08-28)
`selftest.py` refuses to start while a scan is live. Two hours later I read the whole corpus twice
anyway, from `python3 - </` and exits 1 *without* writing the `done`
marker — the right direction. But `chain.run.sh` has no `set -e` and no `||` guard, so that exit
status is discarded and the final `echo "[chain] all done"` runs unconditionally. The marker the
supervisor withholds protects nothing, because the thing waiting downstream waits on the chain's
line, not on the scan's.
That is survivable only because the refusal lives two files further on: `build_page.py` exits
rather than build when any scores file is shorter than its corpus, so a truncated scan cannot reach
the page. `analyse.py` deliberately does *not* refuse — a seeded prefix is a valid paired sample,
which is the whole reason the stages are ordered as they are. **So `analysis.txt` existing is not
evidence that the scan completed**, and `analysis.txt` is precisely the file I read the published
numbers out of. Before authoring any copy: check `[run_scan] / complete: N/N` in each
scan log, not the chain's own summary line. Prompted by the arithmetic in this file being wrong
once already today — 3,408 + 8,839 = 12,247 exactly, and the 40-passage "gap" I thought I saw was
a file line-count from one moment paired with a progress line printed before it.
**2026-08-28 13:18 — hello adversarial scan complete, 12,247/12,247.**
Verified by set equality against the corpus rather than by line count: 12,247 lines, 12,247 unique
ids, zero duplicates, and `comm` against `passages.jsonl` gives missing=0, extra=0. That is the
check `build_page.py` will repeat and the one `run_scan.sh` cannot do — its loop compares `wc -l`
to `wc -l`, which is a weaker statement that happens to coincide here. Worth recording that the
scan survived four OOM kills (attempts resuming at 0, 2,384, 2,456, 2,728 and 3,408) and the
append-JSONL plus skip-set lost nothing and double-scored nothing across all five processes.
Per-stage marker present: `[run_scan] hello/corpus complete: 12247/12247`. Openai adversarial
started 13:18 from zero — the long stage, ~12,247 at 0.66/s.
**2026-08-28 — all four stages verified complete; the completeness check that proved nothing.**
Chain finished. Four markers present: `[run_scan] hello/corpus`, `hello/control`,
`openai/corpus`, `openai/control`, each `complete: N/N`, no `GAVE UP` in any log. Six OOM kills
across the study; every restart resumed from the append-log exactly and lost nothing. All four
score files then checked by set equality against their corpora — 12,247/12,247 twice,
1,287/1,287 twice, zero missing, zero extra, zero duplicates.
That second pair of numbers took two attempts, and the first attempt is the part worth recording.
The id-extraction I used on the adversarial files —
`sed -n 's/.*"n": *\([0-9][0-9]*\).*/\1/p'` — matched **nothing** in either control file, because
control ids are strings (`"n": "#"`, joined from `control.jsonl`'s `id` and `seq`)
while adversarial ids are integers. `comm` then compared two empty sets and reported
`missing=0 extra=0`, which is character-for-character what a passing check looks like. I read it
as a pass.
This is the mirror image of the parse-bug rule already in this file. A parser that fails *closed*
manufactures findings, and I know to look for those because they are flattering. A parser that
fails *open* manufactures **confirmations** — and nothing about the output invites a second look,
because the output is exactly what success looks like. The tell available at the time was a column
I had not printed: the count of ids extracted. Any comparison of two sets must print the size of
both, and a completeness check whose inputs are empty must be a failure rather than a pass.
**2026-08-28 — the tier rule did not fire.** Pre-registered switch: if the strict-minus-loose gap
exceeds 2.00 pp, the book-date-verified tier becomes the headline. Measured: hello **-0.37 pp**
(172/3,751 vs 421/8,496), openai **+1.22 pp** (105/3,751 vs 134/8,496). Both inside the limit and
in opposite directions, so the full corpus stays the headline and both tiers are published in the
breakdown. Recording this because a rule that never fires leaves no trace otherwise, and "the rule
did not fire" and "the rule was never evaluated" have to be distinguishable a year from now.
**2026-08-28 — B3 as written and B3 as enforced are not the same rule.**
B3 reads "every scored passage contains no post-1999 year in its own text". The rule the corpus was
actually drawn with is `merge.py`'s `FUTURE_YEAR = \b(20[0-3]\d)\b` — 2000 to 2039. The build's own
B3 check had been hand-written as `\b(?:19\d\d|20\d\d)\b` with `> 1999`, which covers 2040-2099 too,
and it fired: **12 adversarial passages, 0 control**.
`merge.py`'s docstring asks for exactly this not to happen — "a second copy of the year regex is how
the corpus and a measurement about the corpus start disagreeing" — and `drift.py` obeys it by
importing `FUTURE_YEAR`. `build_page.py` had typed its own. Two copies of one rule, and the drift
landed in the range neither of us was thinking about when the regex was written.
I read all twelve in context before deciding anything. Seven are *Project 2061*, the AAAS
science-literacy programme (one of them in a book titled *Science For All Americans*, which **is** a
Project 2061 report). The rest: an ESRC grant number `ROOO 23 2077`, a fax number `993-2082`, a
logic textbook asking whether "there will be a solar eclipse in Moscow on 27 December 2088" is true
or false, a 1994 projection about the year 2065, and a student writing prompt beginning "The year is
2050". Two of the twelve are not years at all. None is evidence that any of this text postdates
1999, which is the only thing B3 exists to defend.
Resolution, in three parts, because "widen the rule until it passes" is the move this file exists to
prevent:
1. `build_page.py` now **imports** `FUTURE_YEAR` instead of re-typing it. One rule, one copy, and
the check still does real work on the control, which `fetch_control.py` never filtered.
2. The 2040-2099 scan stays, as an enumerated allowlist: `b3_wide_exceptions.json` records all
twelve with the reason each was kept, ships with the data, and **the build refuses to render if
the set differs from that file in either direction**. A thirteenth halts the build and gets read
the same way; a disappearing one halts it too.
3. The limitations bullet said the passage-level rule ran "no exceptions". It now states the real
range, the twelve, and links the file. That sentence was false when I wrote it — not because the
data changed, but because I described the rule from its name.
**2026-08-28 — a qualifier that outlived its data, and an exemption that outlived its condition.**
Two more of the same shape, both caught by guards rather than by reading:
The examples section called the run-to-run batching wobble "far lower" than the gap between the top
three scores, and the build read "far" as an order of magnitude. True at a couple of hundred paired
scores; the padding check kept scoring pairs, the worst wobble grew to 4.29e-06 against a
first-to-third gap of 8e-06, and the build refused. **The right fix was not the threshold.** The
qualifier is now computed — `PAD_RATIO`, printed in the sentence — so the page states the margin
(1.9x) instead of characterising it, and what remains as a refusal is the claim itself: if the
wobble ever reaches the gap, the ordering is inside the noise and the paragraph is wrong rather
than overstated.
`selftest.py` carried two hardcoded sets exempting the seven outcome-dependent copy keys from its
placeholder checks, on the correct grounds that they were deliberately absent. They are written now,
and the exemptions would have gone on forgiving them forever: deleting `TITLE` from `copy.json`, or
dropping a `{{NAME}}` from the template while leaving its key behind, would have passed selftest in
silence and failed only at build time. Both sets are gone; `built` already counts every `copy.json`
key, so the checks are strictly stronger without them. The general form: **an exemption written for
a temporary condition has to be removed as part of satisfying that condition**, because nothing else
will ever prompt it — the exemption's whole job is to keep quiet.