Do AI-image detectors agree about which real photographs are fake?

They mostly do not agree. Three detectors survived the pre-registered quality floor. Between them they call 489 of the 598 real photographs fake at least once — and 308 of those 489 accusations, 63.0%, come from exactly one of the three. Matching all three to the same 5% false positive rate makes it worse, not better: 147 of 172, 85.5%, are one detector's opinion alone, and not one photograph in the corpus is called fake by all three.

Results

What survived

Two detectors failed the pre-registered labelled-AUROC floor of 0.6 and were excluded before any agreement statistic was computed, with their numbers printed rather than quietly dropped: dist at 0.5758 and the unattributed detector (cf) at 0.5207. Both are close enough to chance that their flags would be noise dressed as disagreement.

Three survived, at labelled AUROC (480 real + 540 AI, same round-trip condition):

598 images verified by sha256, 598 scored by all three detectors, 0 dropped. Sieve reproduced the two prior studies' published run exactly — 598 shared images, maximum absolute score difference of zero, zero decision differences, the same 19 flags. That is the halt that says this harness is the one that produced the earlier figures, not a lookalike.

At their shipped decision rules

Each detector read the way a person who downloads it gets it:

Intervals here and below are Wilson score intervals, from dets._measure.wilson.

Those three rates span a factor of more than twenty on identical bytes. aidet calls three quarters of a corpus of EXIF-carrying camera originals fake; its orientation was checked twice (see below), so this is a trigger-happy shipped threshold, not a reversed head.

At least one of the three flags 489 of 598 — 81.77% [78.48, 84.66]. Three independent detectors firing at these same rates would give 83.99%, so the observed union is very slightly below independence: the detectors are, if anything, marginally more spread out than three unrelated coins.

Unanimous accusation — all three flagging the same photograph — happens 10 times in 598: 1.67% [0.91, 3.05].

How many detectors call one photograph fake:

Of the 489 photographs some detector calls fake, 308 (63.0%) are called fake by exactly one and 181 (37.0%) by more than one.

Pairwise, as lift — P(B flags | A flags) divided by B's own base rate, so 1 means no association:

The two third-party detectors — the pair with no conflict of interest — have a lift of 1.00 and a Fisher p of 1. Knowing that aidet flagged your photograph tells you nothing at all about whether sieve will.

FPR-matched at 5%

The shipped rates differ so much that raw overlap is partly a statement about calibration. So the secondary analysis re-sets every threshold to fire on exactly 5% of the labelled real images and repeats everything:

At least one flags 172/598 — 28.76% [25.28, 32.52], against 30.32% if independent. Unanimous: 0 of 598, 0.00% [0.00, 0.64]. Of the 172 accused, 147 (85.5%) by exactly one detector, 25 (14.5%) by more than one.

Matching the rates raises the median lift, from 1.07 to 1.59 — so what association exists is real and was partly hidden by calibration differences. But the two third-party detectors go to lift 0.58: at equal false-positive rates, a photograph aidet calls fake is less likely than baseline to be one sieve calls fake. One image out of 598 is flagged by both.

The conflict of interest, measured

mine is my own published build, and it is in both of the only pairs with a measurable association. That is exactly the pattern a shared backbone would produce, so the two model files were compared rather than assumed (agree-backbone.json):

• different sha256, and neither file is a copy of the other;
• different input normalisation — CLIP statistics for mine, ImageNet statistics for sieve;
• different upstream (buildborderless/CommunityForensics-DeepfakeDet-ViT vs Phineas1500's sieve-ai-image-detector release);
• different calibration (a Platt fit versus a shipped sigmoid bias);
• but the same input size, and parameter counts estimated from file size that differ by 0.14% — 21.86M against 21.89M.

The other thing to say plainly about mine, since nobody else will: at its shipped threshold it calls 36.62% of a corpus of EXIF-carrying camera originals fake — more than ten times Sieve's rate on the identical bytes, and outside the band I pre-registered for every detector. That threshold was fitted to hold up a recall floor across delivery conditions (JPEG, downscale, re-encode), and this is what that choice costs on the other side of the ledger. It is the worst number in this study about any tool I control, it is not an argument for using it, and it is here rather than in a footnote because the alternative is publishing a study about detector disagreement that quietly omits its own author's detector's false-positive rate.

Follow-up, 29 August 2026. That number is now explained rather than merely admitted. A separate pre-registered run scored the same 598 originals three ways — untouched, resampled to a 512px short side, and cropped to 512x512 at native scale — and the resampled arm drops to 13.38%, which is that extension's own eval-set rate. So the gap between the two corpora is the size of the files, not the operating point, and the eval set contains no real image with a short side above 1280px. The write-up, including the mechanism I pre-registered and got wrong, is in the extension's README.

So: not the same weights, but very likely the same architecture class, both fine-tuned for the same task. That is enough to explain a lift of 2.01 without any claim that the two agree about photographs in general, and it is why the honest headline uses the third-party pair — 1.00 shipped, 0.58 matched — as the estimate of how much two unrelated detectors agree.

Pre-registered predictions: 4 of 9 landed

The five that missed, with their measured values, because a pre-registration you may quietly reinterpret is not one:

Four of the five misses are in the same direction: detectors flag real photographs far more often than I predicted, which pushes the union up. The fifth is the interesting one. The pre-registration said:

Pairwise lift, P(B flags | A flags) / P(B flags): > 1 for most pairs, typically 1.5–10. Lift, not raw agreement, is the statistic: raw agreement is mostly a restatement of B's own base rate.

The direction held — 4 of 6 ordered pairs came in above 1, inside its band — but the magnitude did not. A median lift of 1.07 at shipped rules is a rounding error away from independence. Under matching the median reaches 1.59, just inside the predicted band, and it gets there entirely through the two pairs involving my own build.

What it means

If a detector accuses your photograph, the single most likely thing about that accusation is that it is the only one you would get. At shipped rules 63.0% of accusations are unique to one detector; at matched rates, 85.5%. Unanimity is rare (1.67%) at shipped rules and absent (0.00%) at matched ones.

That cuts both ways, and the second way is the uncomfortable one. A second opinion is worth getting, because it usually disagrees. But by the same arithmetic, an accuser who keeps opening tools will eventually find one that flags almost anything: at least one of three detectors calls 81.77% of a corpus of real camera photographs fake.

The question

Three earlier studies of mine asked which real photographs a detector calls fake — glossy product close-ups, tight framing, web recompression — and all three came back null. Each used a single detector.

The question none of them asked: when a detector flags your photograph, would a different detector have flagged it too?

That is the question that decides how much a false positive costs you. If detectors largely agree, a flag is a property of the photograph, and a second opinion tells you nothing new. If they disagree, being called a faker is mostly a function of which tool the accuser happened to open — and the fix for an accusation is to go get another one.

Corpus

598 Wikimedia Commons camera-originals: 298 "glossy" plus 300 control, the same two sets used by the glossy and framing studies. Each image is pinned by sha256 and carries an EXIF camera make/model, so they are real by construction rather than by assumption.

They were re-fetched from the same URLs for this study. Any image whose sha256 did not match the pinned value was dropped and counted, never silently substituted — a substituted image is a different photograph wearing the old one's filename, and it would land in the results as a detector disagreement.

Both arms are pooled. The glossy/control split was the previous studies' treatment variable and came back null; here it serves only as a source of subject diversity.

Condition

Every detector sees identical bytes: a q97 JPEG round-trip of the full-resolution original, produced by importing glossy/measure.py:degrade(edge=None, q=97) rather than re-typing it.

Two reasons it is a round-trip rather than the downloaded file. First, that is the condition that produced the prior studies' numbers, so it is the one their figures can be checked against. Second, and more important for this study: one condition only, so a disagreement between two detectors cannot secretly be a disagreement about which image they saw.

The five detectors

Chosen because they are the ONNX image detectors on this box — not because of anything they scored. Each is read at its own shipped decision rule, because a shipped rule is what a person who downloads the tool actually gets.

Detector 4 is mine. That is a conflict of interest, and it is disclosed everywhere its number appears.

Detector 5's upstream repository is no longer attributable from anything on this box. It is kept and labelled unattributed rather than dropped, because dropping a detector after seeing its numbers filters in the wrong direction — and the agreement question does not need its name.

Orientation, and why it is fixed off-corpus

A two-class detector's "AI" class is a softmax index, and reading the wrong index turns a 3% false-positive rate into a 97% one. A corpus of real photographs cannot tell the difference: both stories predict "the detector flags a lot" or "the detector flags a little" equally well.

So orientation is fixed on the labelled eval set (480 real + 540 AI, same q97 condition), never on this corpus. Two independent checks have to agree before any corpus score is computed:

• the data check — the orientation that gives labelled AUROC > 0.5;
• the documented check — the softmax index the model's own config.json names as its AI class.

check_orientation.py asserts these two agree for every detector. They are not redundant: the data check is only trustworthy when a detector separates the classes well, and one detector here lands close enough to chance that luck could have put it on either side.

Pre-registered halts

A rule nothing computes never fires, so each of these is an assertion in analyse.py, not a note in a document:

• union >= max(individual) and unanimous <= min(individual) — arithmetic. A violation means the flag matrix is wrong, not that something interesting happened.
• every detector's labelled AUROC > 0.6 after the orientation fix; anything below is reported and excluded, with its number printed.
• every detector scored the same set of images. A detector that silently skipped a decode may not be compared against one that did not.
• images scored == images whose sha256 verified.
• the prior studies' Sieve numbers on this corpus (10/298 and 9/300 at 0.65) reproduce exactly. Same corpus, same condition, same imported scoring function — this is a re-run, not a replication, so anything other than those counts means the harness is not the one that produced the published figures.

Predictions are scored separately and are not assertions. A prediction landing outside its band is a result; asserting on it would turn the study's most interesting possible outcome into a crash.

Every number in this write-up is checked against result.json, orientation.json and backbone.json by checkprose.py before publication: unexplained numbers fail, a rate printed beside the wrong detector's name fails, a block quote that is not verbatim fails, and a missed prediction whose value the prose omits fails.

What this cannot show

• Nothing about commercial detectors. Four of the five are open weights and one is Sieve's shipped model; the vendors people are actually accused by are not measured here.
• Nothing about AI images. This is a false-positive study. Recall is not measured, so "detector X flags fewer real photographs" is not "detector X is better".
• Five detectors is not a random sample of detectors. It is the set that runs on this box.
• Three detectors is a small number of pairs. Three pairs, one of which involves my own build, is not enough to say anything about detectors as a population.

Everything behind this

The full result file — every detector's threshold, flag count and interval under both decision rules, all three pairwise cells with their Fisher tests, the overlap histogram, the labelled AUROCs, the reproduction check against the earlier run, and each pre-registered prediction with the band it was written against — is agree-result.json. The pre-registration, unedited since before the first score, is agree-prereg.md; the shared-backbone comparison is agree-backbone.json. The corpus is the one the two subject studies used, published with them: glossy and control, every source URL, camera model and SHA-256.

Nothing above was typed by hand. The prose passes a checker that refuses any number the result file cannot account for, that pins each detector's rate to the detector nearest it so a swapped pair fails, and that fails the document outright if a prediction missed its band and its measured value is nowhere in the results.