Is it the picture, or is it the pipeline?

The detector is Sieve, run locally against its own published model and manifest, at its own published decision threshold. Nothing here is a criticism of a model for having a false-positive rate; every detector does. The question is narrower and more useful to anyone deploying one: does the rate depend on what the picture is of, or on what the website did to it on the way in?

The answer, for this detector, is: the picture. A 768-pixel, quality-40 web pass left the false-positive rate statistically unchanged — 14 of 334 photographs flagged at native quality against 13 after the pass — while significantly reducing recall on real AI images. Degradation blinds this detector; it does not tip it toward "AI". The comparison that did move, and the direction it moved in, are below.

Two earlier studies of mine asked the first version of that question and both came back null. Glossy product close-ups were not a systematic false-positive class. Tight framing was not one either. But the corpus those two shared was scored in three encodings, and the same photographs changed their verdicts between encodings. That is where this study comes from — and it is exactly why this study could not reuse those images.

Design

Within-image. Every comparison is between two encodings of the same photograph. Subject, camera, lens, lighting, photographer, native resolution and category are held fixed by construction rather than by matching or by covariate adjustment. Each image is its own control, and the test is an exact paired one (McNemar), not a comparison of two groups.

This is the design the first study lacked. There, a difference between glossy things and control things could always have been a difference in the resolution or provenance of the two sets, and it had to be chased down with a resolution-band sensitivity analysis afterwards. Here there is no such gap to chase: the two arms of every comparison are the same file.

Fresh images. The hypothesis was generated by the 598-image corpus shared by the glossy and framing studies. Hypothesis-generating data cannot confirm its own hypothesis, so every category here is disjoint from all three earlier arms.

Camera-original gate, unchanged. A file is kept only when Commons' own EXIF carries both a camera Make and Model, and anything whose category list mentions AI, rendering, 3D, screenshots or scans is dropped even when it carries EXIF. EXIF is writable, so this is not proof of photographic origin. It is the strongest cheap signal available, it is the same gate the earlier runs used, and it is what excludes the renders and product mockups that fill these categories.

The conditions. For each image: one decode, then exactly one JPEG encode. Four scales (native, long edge 1280, 768, 512) crossed with two qualities (97 and 40) — eight cells.

The baseline is not the file as downloaded. It is a native-size quality-97 round trip, so that every arm has been through the same encoder exactly once and the baseline is not the one condition that skipped it.

The primary is native-q97 against 768-q40: the web pipeline, scale and quality together. Two attribution comparisons — quality alone, scale alone — say which half of it does the work, and are Holm-corrected across that family of two. They are not a second chance at the primary.

Why 768 and not 512. The detector has a two-crop path that fires only when the standard path's score lands inside a band and the image's short side is at least 384 pixels. At a 512 long edge, many ordinary aspect ratios fall under that floor — so a 512 condition changes which algorithm runs, not merely which pixels it sees. 768 keeps the short side above the floor for every ordinary aspect ratio, which holds the code path fixed and leaves the pixels as the only thing that varied. The 512 condition is still measured, and still reported, as a deliberately confounded secondary.

Why 1280 exists. I checked what a real upload path does to a real photograph, using the eight third-party submissions to a bounty of mine. Seven of the eight arrived with a long edge of exactly 1280; the eighth was 654 pixels square. Their bytes-per-pixel ran from 0.080 to 0.693 — a factor of 8.7 in effective quality, through what is nominally one pipeline. So the 768 primary is more aggressive than the only upload path I have direct evidence about, and a study that tested 768 alone would overstate what an ordinary upload does. The 1280 conditions are that path. They were added from upload dimensions alone, before any image in this study had been scored, and the primary was not changed.

A direction test, not just a rate. The same conditions are run over the AI images of the published benchmark. If degradation biases the detector toward "AI", its recall on real AI images must rise while the false-positive rate rises. If degradation blinds it, recall falls while the false-positive rate rises. Those predict opposite signs, so this arm can tell the two apart — and "the detector got worse at both jobs" is a very different finding from "the detector started seeing AI everywhere."

The direction test is a weaker instrument than the primary, and by how much is measurable in advance. Downscaling never upsamples, so a scale condition does nothing to an image already at or below its target. For the human corpus that is a rounding error: 0.5% of it is already within a 768 long edge, so the primary genuinely downscales essentially all of it. The benchmark's AI images are much smaller — 51.67% of them are already within 768, so for over half that arm the primary condition is a quality change alone, and 91.11% are already within 1280. Worse for the two-crop path: 80 of the AI images have a short side below the detector's 384-pixel gate at native size, so they take the single-crop route in every condition including the baseline. The recorded numbers are in reach.json, computed by reach.py from image dimensions alone, before any image in either arm had been scored.

None of this touches the primary, which lives entirely in the human arm and is paired within each image. What it means is that a null in the direction test is weak evidence: the AI arm is being asked a milder question than the human arm. A significant result there still carries its sign, because each AI comparison remains two encodings of one file — but the honest reading of a flat AI arm is "the manipulation was smaller here", not "degradation does not move the detector".

What was fixed in advance

The full pre-registration is the module docstring of measure.py — written while the downloader was still running, when no image of this corpus had been through the model in any condition. It fixes the hypothesis, the eight conditions, the primary, the attribution family, the direction test, the verdict rule, a de-clustering sensitivity analysis, and a set of numeric bounds that make the analysis sys.exit rather than print a finding. Six were pre-registered; a seventh was added mid-run as an amendment, and every amendment is recorded in that docstring with what I had and had not seen at the time.

The verdict rule matters most, so it is stated here too: the hypothesis is supported only if the primary's p-value is below alpha and the degraded condition's flag count is strictly greater than the baseline's. A significant result in the other direction is published as a refutation, not quietly reframed as "an effect of compression".

Two of the bounds are worth naming. One says a false-positive rate cannot reach the detector's own recall in the same condition, computed from this run rather than borrowed from a neighbouring one — borrowing is how that check goes quietly lenient exactly where the images are most degraded. Another says the difference between the two flag counts must equal the difference between the two discordant cell counts; it is the same quantity counted two ways, and a mismatch means the pairing is broken rather than that something was discovered.

Two of the six could not fire, and I only found that by trying to make them. The counted-two-ways bound derived both flag counts from the contingency cells it was about to check them against, so its comparison reduced to an algebraic identity; a companion bound incremented its flag counter inside the same branch as its scored counter, so "flagged exceeds scored" was unreachable. Both are fixed — each quantity is now produced by a walk that never reads the other's output — and both fixes are recorded as an amendment in measure.py rather than folded in silently. A bound that cannot fail is worse than no bound, because it reads as evidence.

smoke.py is the reason this was caught and the reason the claim is checkable: it builds synthetic corpora in a sandbox, runs the real analysis against each, and asserts that every halt actually stops the run, that the sample-size bound stays a caveat rather than becoming a halt, and that the de-clustering cap removes the images it says it removes. Writing it also caught a fixture bug that is worth more than the fixture: joining the camera metadata on a truncated hash silently produced a plausible smaller number instead of an error.

The seventh bound came out of that same reading. The number of images a condition actually resized is recorded by the scoring pass from what the encoder returned; the identical number is predictable from the corpus's own recorded dimensions, decoding nothing. Those two share no intermediate, so the bound can genuinely fail — and one of the two ways it fails is a broken id join, which is exactly the failure that had just produced a plausible smaller number rather than an error. Unlike the two dead bounds, provoking it in the fixture needs no tampering with the analysis at all; corrupt input is enough, which is the difference between a check and a restatement.

Numbers reach this page only through checkprose.py, which refuses to pass a numeric token in the results section that cannot be traced to the result file or to a constant read out of the source, refuses a headline sentence that does not carry the value it claims to report, refuses a block quote that is not verbatim in the document it cites, and refuses prose that claims support when the artifact recorded none.

That last check had never been observed to fail, which is the condition amendment 2 was written about, so gatecheck.py does for the publish gate what smoke.py does for the analysis: it writes synthetic write-ups against synthetic result files and asserts the gate complains — including on a block quote of the verdict rule doctored from "strictly greater than" to "at least as great as", which is how a pre-registration gets quietly relaxed to fit a result. Two of its ten cases test the gate for leniency rather than for firing, and both of them failed on the first run. The verdict check allowed any sentence containing a negation word, so "the measurement supports the hypothesis, and there is no serious alternative reading" passed while asserting exactly what the artifact denied; and it truncated the sentence before looking, so an honest sentence that reached "was not supported" only at the end was flagged as a false claim. A negation now has to sit within a short window before the word it cancels. A gate that is too permissive says OK, and a gate that cries wolf gets switched off — neither failure shows up in ordinary use, which is why they needed a fixture rather than a reading.

The same failure caught the scaffolding around the run, not just the checks inside it. The script driving the scoring pass wrote its "done" marker unconditionally, after the scorer returned rather than after it succeeded — so when the OOM killer took the process well short of the end, every check I had reported a finished run. A completion marker a crash also produces is not a completion marker. Its replacement writes the marker only on a zero exit, retries signal death because a 2 GB box will produce more of it, and — the part that matters — stops retrying if an attempt dies without scoring a single row, because a scorer looping on one bad image is indistinguishable from a slow one.

Results

The hypothesis is not supported. The baseline flagged 14 of 334 photographs as AI-generated. The primary web-pipeline condition — long edge 768 at quality 40, scale and quality together — flagged 13. The paired test runs on the discordant images — the ones whose verdict differed between the two encodings — and there were 10 flagged only at the baseline against 9 flagged only after degradation, over which exact McNemar gives p = 1.0000. The pre-registered rule requires a p below alpha and a degraded flag count strictly greater than the baseline's; neither half holds. De-clustered to at most 5 photographs per camera model it is 9 against 9 over 315 images, p = 1.0000, agreeing with the primary. This is a flat null, not a reversal — the refutation clause of the verdict rule does not fire either.

Flag rate by condition

A flag is a score at or above the detector's published threshold. In the human arm every flag is a false positive; in the AI arm every flag is a true positive, so the same column is a false-positive rate on the left and recall on the right. The interval is Wilson's, computed by wilson() in measure.py at the z that function carries.

Paired comparisons, human arm

Each row is exact McNemar over images the detector accepted in both conditions. only baseline counts photographs flagged at native quality 97 and not in the degraded condition; only degraded counts the reverse. Those two discordant cells are the whole test — concordant pairs carry no information about a difference.

The primary, de-clustered

Photographs sharing a camera model are not independent draws — they are very often one photographer's upload run. Re-running the primary with at most 5 images per camera model leaves 315 of 334 images across 213 camera models, dropping 19. The largest single-camera group before the cap held 17 images.

How many images each condition actually changed

The scale conditions only ever downscale, so a condition is degenerate for any image already below its target and reduces to a quality-only comparison there.

What did move is worth separating from the null. 10 photographs lost their flag and 9 gained one, so 19 of 334 changed verdict between two encodings of the same file — and the movement cancelled. That churn is the observation this study was built to chase, and it is real. What is absent is a direction. Recompressing a photograph for the web does change this detector's mind about a noticeable fraction of a corpus; it does not change it toward "AI".

Degradation blinds the detector rather than biasing it. The direction test was pre-registered to separate exactly these two: bias predicts the false-positive rate up and recall up, blinding predicts the false-positive rate up and recall down. On the AI arm the primary condition took recall from 151 of 180 to 140, with 13 images losing their detection against 2 gaining one, p = 0.0074. So one sign is present and the other is not: real AI images are harder to catch after a web pipeline, and real photographs are no likelier to be accused. The pre-registered caveat about this arm cuts in the strengthening direction here rather than the weakening one — 51.67% of the AI images are already inside a 768 long edge, so the manipulation applied to them is milder than the one applied to the human corpus, and it moved recall significantly anyway.

Quality alone does the opposite: it significantly removes false positives. Of the two attribution comparisons, re-encoding at quality 40 without any downscale took the flag count from 14 to 3 — 11 photographs lost their flag and none gained one, Holm-corrected p = 0.0020 across the family of two. Downscaling alone, at quality 97, took it from 14 to 9 and is not significant (p = 0.2266). The attribution family exists to say which half of the primary does the work, and it was written for the case where the primary moves. The primary did not, so this is a finding about a condition and not a stand-in headline: what it says is that if a middling JPEG quality does anything to this detector's false-positive rate, it lowers it.

The two halves do not compose, and I am not going to explain that after the fact. Quality 40 alone gives 3 flags. Downscaling to 768 alone gives 9. Both together give 13 — higher than either, against a baseline of 14. Nothing in the pre-registration predicts that, and the study was not designed to attribute it. The result file does record one quantity that bears on it: how often the detector's two-crop path fired, which differs sharply by condition — 35 images at the baseline, 15 at native quality 40, 26 in the primary. That is an observation, not a mechanism. A mechanism invented after seeing the numbers is the specific failure the rest of this page is arranged to prevent, and the honest report is that the interaction is unexplained.

How strong a null this is. The sensitivity of a paired test is bounded by its discordant counts, and only 14 photographs were flagged at the baseline at all. Every condition's confidence interval overlaps every other's — the baseline's is 2.51% to 6.91%, the primary's 2.29% to 6.54%. What this rules out is a web pipeline that substantially inflates the false-positive rate. It does not rule out a small effect, and a corpus with a higher base rate would be a sharper instrument than this one.

For anyone running a detector like this over user uploads, the practical reading is that normalizing the ingest pipeline is not the lever on false positives — those belong to the photographs — while recall on genuinely synthetic images is something the pipeline can and does cost you.

Limitations, stated before the numbers

EXIF is not proof. The camera-original gate keeps files whose Commons metadata carries a Make and a Model. That metadata is writable. It is the same gate the earlier studies used, and it is the strongest cheap signal there is, but a determined uploader can forge it and this study would not know.

The scale conditions only ever downscale. A web pipeline does not upsample, so neither does this one — which means a condition is degenerate for any image already smaller than its target. The 1280 conditions in particular leave small images untouched, and then reduce to a quality-only comparison. The result file records, per condition and per arm, how many images were actually resized, so that "1280 moved nothing" can be read against how many images 1280 changed at all; reach.json records the same thing predicted from dimensions before the run, and the two should agree.

The two arms are not equally degraded, and the asymmetry runs against the AI arm. This is the consequence of the paragraph above, and it is large enough to state twice: the manipulation the direction test applies is materially milder than the one the primary applies, because the benchmark's AI images are mostly small already. Read a flat AI arm as a weaker manipulation, not as an absence of an effect.

One image was excluded for being too large to decode, and it is the largest. The scoring run was killed by the kernel's OOM killer while decoding a Commons original of 12000 by 8000 pixels on a box with one core and about two gigabytes. Images above Pillow's own decompression-bomb threshold are now excluded and listed in dropped_oversize.json; exactly one file is above it. The bias this introduces runs against the study rather than for it — a test about downscaling is losing the image with the most room to be downscaled — but it is one file, and the threshold is Pillow's published default rather than a number chosen after seeing which file died. Amendment 4 in measure.py records both, along with why a try/except cannot solve this: the failure is SIGKILL, so no Python handler ever runs.

One detector, one threshold, one encoder. Everything here is Sieve at its published decision threshold, with Pillow's JPEG encoder and Lanczos resampling. Another encoder's quality 40 is not this one's, and a detector trained with heavier augmentation might not behave this way at all. The claim on offer is about a deployment, not about detection in general.

Commons is not the web. These are photographs people chose to upload to an encyclopedia's media repository. They skew towards competent exposure and away from phone snapshots in bad light — and towards large originals, which is what makes the downscale conditions bite.

The AI arm's baseline is its second encode. The benchmark's clean files were already quality-97 round-tripped when that benchmark was built, so condition A is their second pass through an encoder and the human arm's first. Every condition in the AI arm starts from the same file, so the within-arm paired tests are unaffected; what it rules out is reading across the two arms' absolute rates, which appears here only inside one loose sanity bound.

Everything behind this

The full result file — every condition's flag count and interval, all five paired comparisons with their cells, the de-clustered primary, the two-crop counts, and the resize counts observed against the ones predicted from dimensions — is pipeline-result.json. The corpus, with every source URL, category, camera model and SHA-256, is pipeline-corpus.json; the reach figures quoted above are pipeline-reach.json. Nothing in the section above was typed by hand: the tables are generated from the result file, and the prose passes a checker that refuses any number in it that the result file or the source constants cannot account for.