# Pre-registration — do five AI-image detectors agree about which real photographs are fake? Written 2026-08-29, before any detector has been run on the corpus. Nothing below may be edited after the first score lands; corrections go in a dated block at the bottom. ## The question Three published studies of mine asked *which real photographs* a detector calls fake, and all three came back null (glossy close-ups, tight framing, web recompression). Each used one detector. The question none of them asked: when a detector flags your photograph, **would a different detector have flagged it too?** This matters because the accusation is what harms someone. If detectors disagree, then being called a faker is mostly a function of which tool the accuser happened to open. ## Corpus The 598 Wikimedia Commons camera-originals already used in the glossy and framing studies: 298 "glossy" (`corpus_glossy.json`) + 300 control (`corpus_control.json`), each pinned by sha256 and carrying an EXIF camera make/model. They are real by construction. Re-fetched from the same URLs; **any image whose sha256 does not match is dropped and counted**, never silently substituted. Both arms are pooled. The glossy/control split was the previous studies' treatment variable and is null; here it is only a source of diversity. ## Detectors — fixed now, before any of them is scored on this corpus Chosen because they are the ONNX image detectors on this box, not because of anything they scored. Each is read at **its own shipped decision rule**, because that is what a person running the tool actually gets: 1. `capcheck/ai-image-detection` — 2-logit softmax, argmax (i.e. P(FAKE) >= 0.5) 2. `jacoballessio/ai-image-detect-distilled` — 2-logit softmax, argmax 3. Sieve's `ft44s` (manifest ft44s-2026-08-19) — its shipped 0.65 4. My own published browser extension build — views `native` + `squash`, mean in **probability** space, Platt calibration a=0.464414 b=3.2143, threshold 0.65. This one is mine; that is a conflict of interest and is disclosed wherever its number appears. 5. A fourth third-party ONNX detector (384px single-logit ViT, CLIP normalisation) whose upstream repository I can no longer attribute from anything on this box. Kept, and labelled unattributed, because dropping a detector *after* seeing its numbers is the wrong direction to filter — and the agreement question does not need its name. **Orientation is fixed on the labelled eval set** (`aidetect/data`, 480 real + 540 AI), never inferred from this corpus. A real-photograph-only corpus cannot tell a 3% false-positive rate from a 97% one with the head reversed. ## Predictions - **Per-detector FPR at its shipped rule: 1–30%.** Sieve (detector 3) measured 3.36% on the glossy arm and 3.00% on the control arm of this exact corpus in two prior studies, so it should land there again; a large move means the harness differs from the one those studies used, not that the world changed. Above 60% means the orientation is flipped. Check the labelled AUROC before believing it. - **Union (>= 1 of 5 flags a photo): 15–50%.** Five independent detectors at ~5% each would give 22.6%. - **Unanimous (all 5 flag): 0–2%.** - **Pairwise lift, P(B flags | A flags) / P(B flags): > 1 for most pairs, typically 1.5–10.** Lift, not raw agreement, is the statistic: raw agreement is mostly a restatement of B's own base rate. - **FPR-matched secondary:** thresholds re-set so every detector fires on exactly 5% of the *labelled* real images, then the whole analysis repeated. If the pairwise lift survives matching, the disagreement is not an artefact of differing calibration. Predict union 15–23% and lift still > 1. ## Halts — the build fails, it does not just warn A rule nothing computes never fires, so each of these is an assertion in `analyse.py`: - `union >= max(individual)` — arithmetic; a violation means the flag matrix is wrong. - `unanimous <= min(individual)` — same. - every detector's AUROC on the labelled eval set > 0.6 after the orientation fix; any detector below that is reported and excluded, with its number printed. - every detector has a score for exactly the same set of images; a detector that silently skipped a decode may not be compared against one that did not. - image count scored == image count whose sha256 verified. - the two prior studies' numbers for **detector 3 (Sieve)** on this corpus (3.36% glossy / 3.00% control at 0.65) must reproduce **exactly** — same corpus, same condition, same imported scoring function, so anything other than 10/298 and 9/300 means the harness is not the one that produced them. ## Condition — pinned before scoring The prior studies did not score the downloaded file. Their `clean` condition is a q97 JPEG round-trip of the full-resolution original (`glossy/measure.py:degrade(edge=None, q=97)`), and that is what produced 3.36% / 3.00%. **Every detector here sees the identical `clean` bytes**, produced by importing that same function rather than re-typing it. One condition only, so a disagreement between two detectors cannot be a disagreement about which image they saw. ## What this cannot show - Nothing about *commercial* detectors. Four of these five are open weights; Sieve's is the one shipped model, and the vendors people are actually accused by (Hive, Optic, Winston) are not measured here. - Nothing about AI images. This is a false-positive study; recall is not measured. - Five detectors is not a random sample of detectors. It is the set that runs on this box. ## Corrections **2026-08-29, before any score existed.** As first drafted, this document attributed the prior studies' 3.36% / 3.00% to *detector 4* (my own extension build). That is wrong: `glossy/measure.py` imports `score_image` from `sieve-test/score.py`, so those numbers are **Sieve's**, at Sieve's 0.65. Two places corrected — the per-detector prediction and the reproduction halt — and the halt tightened from "within 1 point" to exact counts, because the same function over the same corpus under the same condition is a re-run, not a replication. My own build has never been measured on this corpus, so it gets no anchor and no prediction beyond the 1–30% band. The `Condition` section above was added at the same time, for the same reason: I had not written down that the anchor is a q97 round-trip rather than the file as downloaded. **2026-08-29, also before any score existed.** The labelled eval set gets the same q97 clean round-trip as the corpus. The pre-registration did not say either way. The reason to say so now rather than after seeing a number: the FPR-matched secondary sets each detector's threshold at 5% of the *labelled* real images and then applies that threshold to *corpus* scores, so if the two sets went through different conditions the match would be against a different distribution than the one it is used on.