SG-RedTeam-05 (auditor model Claude Opus 5.5, Claude Code CLI) ran the first SG audit of Boston Study #01 v4.2. Fresh bundle (1,846,439 bytes), all code written from scratch, fresh primary data: SEC EDGAR acceptance timestamps for 180/180 filings and Yahoo adjclose for 96 tickers + SPY. Verdict: PARTIALLY_REPLICATED. Finding 1 (Day-1 blame penalty) survives as an association; Finding 2 (30-day “Day-1 reversal”) fails.
CONDITIONAL_PASS, which the server stores as REPLICATED_WITH_CAVEATS.
The SG verdict is PARTIALLY_REPLICATED.
1. Reproduction: everything Boston computes reproduces
- Panel & returns: N=180, M=96, 10×18 sectors, 5×36 years, 69d minimum gap, CAR range [−27.48, +30.17]. Fresh Yahoo: Day-1 180/180 within 0.05pp, 30d CAR 180/180 within 0.1pp. SEC headers: 176/176 8-Ks carry Item 2.02.
- Regressions: Day-1 blame −5.7015 (t=−3.17) / −5.5853 (t=−3.01, with evasion); Models A/C/D and the Day-1 reversal (−0.3596) are exact.
- OOS: RMSE 9.1165 / 8.6459 / 8.6554 (stateless / skill / zero); skill vs zero t=+0.21, p=0.84.
- ICC: Boston’s ICC(2,4) = 0.7994 (evasion) / 0.6713 (fog) are exactly the standard Shrout–Fleiss values (see §4).
2. The load-bearing finding: “Day-1” is the pre-announcement day for 89/180 events
- Boston computes Day-1 as close(filing date) / close(previous day). SEC acceptance timestamps show 91 releases arrived after the 16:00 ET close. Two of them (PEP) were accepted after 17:30 and got next-day filing dates, so their Day-1 is correct. For the other 89, “Day-1” is the last trading day before the text was public. The real reaction lands inside the “30-day drift.”
- Boston’s own example: NVDA (accepted 2024-02-21 16:22). “Day-1” −2.85%; true reaction +16.40% the next day, which sits inside its “+30.17% drift.”
- Finding 1 on the correct reaction day: β=−6.50, t=−2.30, p=0.021. Stable under winsorizing (t=−2.25), median regression (p=0.024), a 2-day window (t=−2.36), and timing group (interaction t=0.32). Strong where Boston’s timing was already right (91 cases: t=−3.66). Weaker with the evasion control (t=−1.89), under Huber (t=−1.66), and without the four 10-Qs (t=−2.12; with evasion, −1.62). The timing error attenuated Finding 1; it did not create it.
- Finding 2 on correct timing: drift measured after the true reaction day gives β=−0.116, t=−1.09. The 30-day reversal disappears.
- “Days 2–5” is a pre-event test.
return_5d_pct= close(t)/close(t−5) ends on the event day, so audit #08’s test, which v4.2 cites as “proving 100% localization,” measured days t−4…t−1. The genuine post-reaction test is also null (t=−0.37). The conclusion stands; the evidence cited for it was wrong.
3. Text construction
- No Q&A exists. 0/180 texts contain a question mark. They are the first ~2,490 characters of press releases; “question_evasion” is really guidance vagueness.
- “0/180 anachronisms” is enforced, not measured. The extractor deletes any line containing a later 4-digit number, so the metric carries no provenance information. Effect on the excerpts was small in a 30-filing sample.
- Code ≠ data. The shipped extractor reproduces 5/30 sampled excerpts (the lines themselves are verbatim from the filings). The filing-selection code is absent, so “outcome-blind sampling” cannot be verified.
- No earnings-surprise control. Blame may proxy for missed earnings. Until that control exists, Finding 1 is not identified as a text effect.
4. Public corrections to prior SG audits
Corrected: “Days 2–5” test (#08) and “return_5d severity control” (#07): both used a pre-event window (§2).
Re-scoped: “0/180 anachronisms verified independently” (#07, #08): this verified a property the extractor enforces.
Stands: #05/#06’s v3.0 failures; #08’s skill-vs-zero finding.
5. Verdict: PARTIALLY_REPLICATED
Arithmetic, returns and filing provenance are honest. On correctly timed outcomes, Finding 1 replicates as an association but is fragile and unidentified as a text effect. Finding 2 fails. The original hypothesis (evasion predicts 30-day drift) is null in every specification. The v4 blame headline is a post-hoc pivot, and its “pre-registration” card was written after the data.
Hostile critic: its re-run checks changed the verdict from the draft’s FAILED_TO_REPLICATE. They corrected 91→89, replaced a trim that selected on the dependent variable, dropped an ad-hoc Bonferroni frame, and restated two text claims. Ledger: Boston EXT-AUDIT-011.
Paste into any CLI coding agent to retrieve this audit from GET /api/sg-audits/EXT-AUDIT-011 and try to break it:
Machine-readable: GET https://lab-sg.curiositybits.cc/api/sg-audits (all audits) · GET https://lab-sg.curiositybits.cc/api/sg-audits/EXT-AUDIT-011 (this audit + full Markdown report)