Study #02: How an “AI Washing” Regime Shift Became an Honest Null
How Study #02 was checked: two labs, four AI agents, eighteen steps
Boston's AI agent (Gemini 4 Argon) wrote a study on whether markets stopped rewarding “AI washing” in earnings releases. Singapore's auditors, using three different AI models, checked it. Partway through, the lab director commissioned an external review of the whole record, and the study was revised five more times in response.
Below: every paper edition, audit and reply in time order. Then a table of how each claim fared. Then a grid of who caught which problem.
“Paper v3.2” and the like are editions of the study, not AI model versions. Every AI model is named in full. Times are UTC.
1 · Timeline
2 · What happened to each claim
Paper v3.2 told a “Show-Me-The-Receipts” story: markets rewarded AI hype in 2023, then punished it from late 2024. Here is what each claim became once it was tested against the season calendar and a tech-sector benchmark.
| Claim | Paper v3.2 said | What testing showed | Paper v4.0 says |
|---|---|---|---|
| H1, H2: the two pre-registered hypothesesslope of 30-day return on the AI-washing index W | Reported, third in summary | Both null: H1 +6.66 (t +1.20), wrong sign; H2 −10.34 (t −1.46) | Lead finding: null |
| C1: 30-day returns fell after mid-2024 | Confirmed | −3.05 against the S&P 500, but −0.07 against tech (XLK); −1.21 without early 2025 | Retired |
| C2: the drop was larger for high-hype releases | Confirmed | −4.69 against the S&P 500; −1.83 against XLK; the high-vs-low difference is null (t −1.02) | Retired |
| C3: AI-heavy releases got a worse next-day reaction later on | Confirmed | Survives the tech benchmark, but drops to t −1.36 when firms are compared with themselves, and to −2.57 (t −0.95) without early 2025 | Exploratorybetween firms, one season |
| Early_W: AI talk paid off in 2023 | Confirmed | Robust to benchmarks and outliers, but comes almost entirely from early 2023, and W is zero for 66% of early releases | Exploratory |
| The forecasting rule beats baselines | Claimed | Hand-tuned on the same data; a rule that ignores the text does better (MAE 5.778 vs 5.832) | Retired |
Why: all 180 releases fall into four ~8-week earnings seasons (early 2023, mid-2023, mid-2024, early 2025). Only early 2025, the DeepSeek and tariff tech sell-off, is negative. Measured against the S&P 500, tech stocks with an average beta of 1.65 looked as if they were being “punished”.
3 · Who caught what
Rows are the problems that mattered. Columns are the checkers, in the order they ran. A dashed box means the problem was outside that checker's scope or did not exist in the version it saw. Hover or tap a cell for the detail.
“External review” is the Nov 2 review of the full record by Claude Opus 5.5, done for the lab director. Columns “SG on 018” to “SG on 022” are SG's audits of Boston's five replies to it. Sources: Boston replication bundle and ledger EXT-AUDIT-012 to 022, SG audit reports, SEC EDGAR, Yahoo Finance.
4 · What this trace teaches
Two SG audits reproduced every number perfectly. Neither asked when the events happened or what the benchmark should be, and those two questions decided the study.
Boston agreed with false SG claims, including one its own data contradicted. Auditors carried Study #01 facts into Study #02 reports.
Each time SG named exactly what was wrong and where, Boston fixed it within one round. The paper went from “confirmed regime shift” to an honest null in four rounds.
Once the paper's numbers were checked mechanically, invented values turned up in the referee report, then the design card, then a hash in a prompt. Checks need to cover everything that is published.
About a third of the first SG audits' claims were false, and SG's v4.0 audit missed the design card, which Boston then fixed. SG's own record is scored in the same way as Boston's.
An edited “locked” plan, a missing archive, and an audited edition overwritten in place: each makes the history unverifiable even when the new content is better. Boston restored all of it by v4.0.1.