============================================================================== hello COMPLETE ============================================================================== adversarial corpus, P(AI)>0.5: 593 / 12247 4.84% [4.48-5.24] ** ceiling on these detectors' error, NOT a rate for 1990s books ** clustered by book (1809 books, 6.8 passages each): [4.26-5.44] 1.55x the passage-level width dispersion across books with 2+ scored (1594 books, 7.5 passages each): 2.47x binomial quantiles: p50=0.0011 p75=0.0147 p90=0.1674 p95=0.3665 p99=0.9952 sweep: >0.50: 4.84% >0.70: 4.70% >0.90: 4.11% >0.95: 3.41% >0.99: 1.67% within 0.01 of each threshold: 0.50:1 0.70:1 0.90:18 0.95:58 0.99:298 batching check, adversarial: 526 re-scored singly, largest |delta| 4.29e-06, verdict flips 0 batching check, control: 168 re-scored singly, largest |delta| 4.98e-07, verdict flips 0 date verification n books flagged rate [95% CI passages] [95% CI by book] loose (catalogue year only) 8496 1201 421 4.96% [4.51-5.44] [4.27-5.69] strict (book-date-verified) 3751 615 172 4.59% [3.96-5.30] [3.63-5.61] TIER GAP (strict minus loose): -0.37 pp [PREREG: strict becomes the headline above 2.00 pp] source query family n books flagged rate [95% CI passages] [95% CI by book] Beginner computer tutorials — Teach Yourself, For Dummies, step-by-step 1113 154 181 16.26% [14.21-18.55] [12.52-20.16] Composition and rhetoric textbooks, reference and encyclopedia entries 1825 295 60 3.29% [2.56-4.21] [2.30-4.36] Management, self-improvement, popular science 2051 266 53 2.58% [1.98-3.36] [1.85-3.38] Mid-1990s popular introductions to the internet and computers 199 39 16 8.04% [5.01-12.66] [1.60-16.15] Nursing, counselling, social work and education textbooks 337 39 16 4.75% [2.94-7.57] [1.80-8.62] Programmed instruction, technical/maintenance manuals, translated textbooks 576 97 18 3.12% [1.99-4.89] [1.57-4.97] Self-help and advisory writing — how-to, career, study skills 1319 232 71 5.38% [4.29-6.74] [3.90-7.04] Self-study textbooks, study guides, ESL instructional materials 3757 565 153 4.07% [3.49-4.75] [3.20-4.96] Translated / non-native-English textbooks — Mir, Tata McGraw, Progress 120 12 2 1.67% [0.46-5.87] [0.00-5.00] US military training series — NEETS, rate training, correspondence courses 950 118 23 2.42% [1.62-3.61] [1.17-4.00] passage length n books flagged rate [95% CI passages] [95% CI by book] 180-239 words 1 1 0 0.00% [0.00-79.35] [0.00-100.00] 240-299 words 770 530 34 4.42% [3.18-6.11] [2.89-6.00] 300-359 words 11448 1789 559 4.88% [4.50-5.29] [4.31-5.49] 360-419 words 28 27 0 0.00% [0.00-12.06] [0.00-0.00] truncated at 512 tokens: 30 of 12247 scored (0.24%) flagged 0, 0.00% [0.00-11.35] OCR damage n books flagged rate [95% CI passages] [95% CI by book] 0-4% out-of-dictionary 9085 1676 529 5.82% [5.36-6.32] [5.13-6.55] 10-14% out-of-dictionary 449 281 6 1.34% [0.61-2.88] [0.23-2.66] 15-19% out-of-dictionary 86 57 1 1.16% [0.21-6.30] [0.00-3.85] 20%+ out-of-dictionary 50 36 0 0.00% [0.00-7.14] [0.00-0.00] 5-9% out-of-dictionary 2577 985 57 2.21% [1.71-2.85] [1.52-2.96] CONTROL (genre-neutral draw, same archive/era/extraction): control corpus, P(AI)>0.5: 32 / 1287 2.49% [1.77-3.49] clustered by book (212 books, 6.1 passages each): [1.49-3.69] 1.28x the passage-level width this is the number that describes 1990s books; the one above describes the detectors sweep: >0.50: 2.49% >0.70: 2.41% >0.90: 1.94% >0.95: 1.55% >0.99: 0.39% DIFFERENCE (adversarial - control): +2.36 pp 95% [+1.06, +3.62] excluding the 4 books in both corpora: adv 4.85% ctl 2.52% +2.33 pp 95% [+1.09, +3.54] same text scored in both corpora: 19 passages, largest delta 0, verdict flips 0 ERA-MATCHED (1990+ only, 10384 adv / 471 ctl): adv 5.26% [4.85-5.70] ctl 2.34% [1.31-4.13] DIFFERENCE, era-matched: +2.92 pp 95% [+1.43, +4.41] decade, adversarial n books flagged rate [95% CI passages] [95% CI by book] 1940s 2 1 0 0.00% [0.00-65.76] [0.00-100.00] 1960s 28 2 0 0.00% [0.00-12.06] [0.00-0.00] 1970s 142 17 2 1.41% [0.39-4.99] [0.00-3.73] 1980s 1691 237 45 2.66% [1.99-3.54] [1.73-3.66] 1990s 9725 1466 529 5.44% [5.01-5.91] [4.71-6.16] 2000s 659 86 17 2.58% [1.62-4.09] [1.17-4.15] (the 2000s bucket is 2000 only, not a decade: nothing in this corpus is catalogued later than 2000) decade, control n books flagged rate [95% CI passages] [95% CI by book] 1970s 409 65 5 1.22% [0.52-2.83] [0.26-2.36] 1980s 407 67 16 3.93% [2.43-6.29] [1.46-6.92] 1990s 460 78 11 2.39% [1.34-4.23] [1.13-3.91] 2000s 11 2 0 0.00% [0.00-25.88] [0.00-0.00] (the 2000s bucket is 2000 only, not a decade: nothing in this corpus is catalogued later than 2000) ============================================================================== openai COMPLETE ============================================================================== adversarial corpus, P(AI)>0.5: 239 / 12247 1.95% [1.72-2.21] ** ceiling on these detectors' error, NOT a rate for 1990s books ** clustered by book (1809 books, 6.8 passages each): [1.66-2.23] 1.16x the passage-level width dispersion across books with 2+ scored (1594 books, 7.5 passages each): 1.37x binomial quantiles: p50=0.0002 p75=0.0004 p90=0.0072 p95=0.0648 p99=0.9067 sweep: >0.50: 1.95% >0.70: 1.50% >0.90: 1.08% >0.95: 0.74% >0.99: 0.45% within 0.01 of each threshold: 0.50:6 0.70:3 0.90:15 0.95:10 0.99:72 batching check, adversarial: 256 re-scored singly, largest |delta| 2.61e-06, verdict flips 0 batching check, control: 158 re-scored singly, largest |delta| 1.92e-06, verdict flips 0 date verification n books flagged rate [95% CI passages] [95% CI by book] loose (catalogue year only) 8496 1201 134 1.58% [1.33-1.86] [1.28-1.89] strict (book-date-verified) 3751 615 105 2.80% [2.32-3.38] [2.19-3.42] TIER GAP (strict minus loose): +1.22 pp [PREREG: strict becomes the headline above 2.00 pp] source query family n books flagged rate [95% CI passages] [95% CI by book] Beginner computer tutorials — Teach Yourself, For Dummies, step-by-step 1113 154 20 1.80% [1.17-2.76] [0.87-2.83] Composition and rhetoric textbooks, reference and encyclopedia entries 1825 295 52 2.85% [2.18-3.72] [1.95-3.80] Management, self-improvement, popular science 2051 266 53 2.58% [1.98-3.36] [1.80-3.43] Mid-1990s popular introductions to the internet and computers 199 39 7 3.52% [1.71-7.08] [1.02-6.37] Nursing, counselling, social work and education textbooks 337 39 8 2.37% [1.21-4.61] [0.97-4.04] Programmed instruction, technical/maintenance manuals, translated textbooks 576 97 5 0.87% [0.37-2.02] [0.16-1.89] Self-help and advisory writing — how-to, career, study skills 1319 232 36 2.73% [1.98-3.76] [1.78-3.80] Self-study textbooks, study guides, ESL instructional materials 3757 565 47 1.25% [0.94-1.66] [0.87-1.66] Translated / non-native-English textbooks — Mir, Tata McGraw, Progress 120 12 1 0.83% [0.15-4.57] [0.00-2.50] US military training series — NEETS, rate training, correspondence courses 950 118 10 1.05% [0.57-1.93] [0.43-1.75] passage length n books flagged rate [95% CI passages] [95% CI by book] 180-239 words 1 1 0 0.00% [0.00-79.35] [0.00-100.00] 240-299 words 770 530 11 1.43% [0.80-2.54] [0.64-2.25] 300-359 words 11448 1789 227 1.98% [1.74-2.25] [1.72-2.28] 360-419 words 28 27 1 3.57% [0.63-17.71] [0.00-11.11] truncated at 512 tokens: 30 of 12247 scored (0.24%) flagged 0, 0.00% [0.00-11.35] OCR damage n books flagged rate [95% CI passages] [95% CI by book] 0-4% out-of-dictionary 9085 1676 199 2.19% [1.91-2.51] [1.85-2.53] 10-14% out-of-dictionary 449 281 11 2.45% [1.37-4.33] [1.12-3.96] 15-19% out-of-dictionary 86 57 2 2.33% [0.64-8.09] [0.00-6.02] 20%+ out-of-dictionary 50 36 0 0.00% [0.00-7.14] [0.00-0.00] 5-9% out-of-dictionary 2577 985 27 1.05% [0.72-1.52] [0.65-1.47] CONTROL (genre-neutral draw, same archive/era/extraction): control corpus, P(AI)>0.5: 21 / 1287 1.63% [1.07-2.48] clustered by book (212 books, 6.1 passages each): [0.97-2.31] 0.95x the passage-level width this is the number that describes 1990s books; the one above describes the detectors sweep: >0.50: 1.63% >0.70: 1.32% >0.90: 0.93% >0.95: 0.62% >0.99: 0.23% DIFFERENCE (adversarial - control): +0.32 pp 95% [-0.45, +1.02] excluding the 4 books in both corpora: adv 1.95% ctl 1.66% +0.30 pp 95% [-0.48, +1.04] same text scored in both corpora: 19 passages, largest delta 0, verdict flips 0 ERA-MATCHED (1990+ only, 10384 adv / 471 ctl): adv 2.08% [1.82-2.37] ctl 2.34% [1.31-4.13] DIFFERENCE, era-matched: -0.26 pp 95% [-1.77, +1.01] decade, adversarial n books flagged rate [95% CI passages] [95% CI by book] 1940s 2 1 0 0.00% [0.00-65.76] [0.00-100.00] 1960s 28 2 0 0.00% [0.00-12.06] [0.00-0.00] 1970s 142 17 1 0.70% [0.12-3.88] [0.00-2.05] 1980s 1691 237 22 1.30% [0.86-1.96] [0.72-1.93] 1990s 9725 1466 210 2.16% [1.89-2.47] [1.84-2.51] 2000s 659 86 6 0.91% [0.42-1.97] [0.16-1.79] (the 2000s bucket is 2000 only, not a decade: nothing in this corpus is catalogued later than 2000) decade, control n books flagged rate [95% CI passages] [95% CI by book] 1970s 409 65 6 1.47% [0.67-3.16] [0.48-2.76] 1980s 407 67 4 0.98% [0.38-2.50] [0.23-2.13] 1990s 460 78 10 2.17% [1.19-3.96] [0.90-3.49] 2000s 11 2 1 9.09% [1.62-37.74] [0.00-10.00] (the 2000s bucket is 2000 only, not a decade: nothing in this corpus is catalogued later than 2000)