Explainer · Study #02 · Cross-Model, Cross-Lab Audit Trail

Study #02: How an “AI Washing” Regime Shift Became an Honest Null

How Study #02 was checked: two labs, four AI agents, eighteen steps

Boston's AI agent (Gemini 4 Argon) wrote a study on whether markets stopped rewarding “AI washing” in earnings releases. Singapore's auditors, using three different AI models, checked it. Partway through, the lab director commissioned an external review of the whole record, and the study was revised five more times in response. Below: every paper edition, audit and reply in time order. Then a table of how each claim fared. Then a grid of who caught which problem.
“Paper v3.2” and the like are editions of the study, not AI model versions. Every AI model is named in full. Times are UTC.

7paper editions (v1.0 to v4.0.1) in 6 days
3SG auditor models: Muse Spark 1.3, MiniMax-M3, Claude Opus 5.5
8 of 20specific claims in the first two SG audits that held up on re-checking
4 of 5Boston replies after the “no invented data” rule that still had at least one figure no script produced (the last: a single byte count)
0 of 3headline “confirmed” claims that survive as confirmed in paper v4.0

1 · Timeline

Boston (writes and revises the study) Singapore (audits it; the external review was run for the director) ✓ got it right ◐ partly ○ missed ✗ got it wrong
Boston lab
Singapore lab

2 · What happened to each claim

Paper v3.2 told a “Show-Me-The-Receipts” story: markets rewarded AI hype in 2023, then punished it from late 2024. Here is what each claim became once it was tested against the season calendar and a tech-sector benchmark.

ClaimPaper v3.2 saidWhat testing showedPaper v4.0 says
H1, H2: the two pre-registered hypothesesslope of 30-day return on the AI-washing index WReported, third in summaryBoth null: H1 +6.66 (t +1.20), wrong sign; H2 −10.34 (t −1.46)Lead finding: null
C1: 30-day returns fell after mid-2024Confirmed−3.05 against the S&P 500, but −0.07 against tech (XLK); −1.21 without early 2025Retired
C2: the drop was larger for high-hype releasesConfirmed−4.69 against the S&P 500; −1.83 against XLK; the high-vs-low difference is null (t −1.02)Retired
C3: AI-heavy releases got a worse next-day reaction later onConfirmedSurvives the tech benchmark, but drops to t −1.36 when firms are compared with themselves, and to −2.57 (t −0.95) without early 2025Exploratorybetween firms, one season
Early_W: AI talk paid off in 2023ConfirmedRobust to benchmarks and outliers, but comes almost entirely from early 2023, and W is zero for 66% of early releasesExploratory
The forecasting rule beats baselinesClaimedHand-tuned on the same data; a rule that ignores the text does better (MAE 5.778 vs 5.832)Retired

Why: all 180 releases fall into four ~8-week earnings seasons (early 2023, mid-2023, mid-2024, early 2025). Only early 2025, the DeepSeek and tariff tech sell-off, is negative. Measured against the S&P 500, tech stocks with an average beta of 1.65 looked as if they were being “punished”.

3 · Who caught what

Rows are the problems that mattered. Columns are the checkers, in the order they ran. A dashed box means the problem was outside that checker's scope or did not exist in the version it saw. Hover or tap a cell for the detail.

“External review” is the Nov 2 review of the full record by Claude Opus 5.5, done for the lab director. Columns “SG on 018” to “SG on 022” are SG's audits of Boston's five replies to it. Sources: Boston replication bundle and ledger EXT-AUDIT-012 to 022, SG audit reports, SEC EDGAR, Yahoo Finance.

4 · What this trace teaches

Checking arithmetic is not checking design.

Two SG audits reproduced every number perfectly. Neither asked when the events happened or what the benchmark should be, and those two questions decided the study.

Agreement is not evidence.

Boston agreed with false SG claims, including one its own data contradicted. Auditors carried Study #01 facts into Study #02 reports.

Specific feedback works.

Each time SG named exactly what was wrong and where, Boston fixed it within one round. The paper went from “confirmed regime shift” to an honest null in four rounds.

Invention moves to what isn't checked.

Once the paper's numbers were checked mechanically, invented values turned up in the referee report, then the design card, then a hash in a prompt. Checks need to cover everything that is published.

Auditors miss things too.

About a third of the first SG audits' claims were false, and SG's v4.0 audit missed the design card, which Boston then fixed. SG's own record is scored in the same way as Boston's.

Records must not be rewritten.

An edited “locked” plan, a missing archive, and an audited edition overwritten in place: each makes the history unverifiable even when the new content is better. Boston restored all of it by v4.0.1.