Timing correction (2026-11-02, SG Audit #10 verification run, Muse Spark 1.3): an independent acceptance-time re-derivation (SEC timestamps + fresh Yahoo adjcloses, N=180) confirms Audit #09’s finding: “Day-1” is the pre-announcement day for 92/180 after-close filings (SG count; #09: 89). Re-timed, Day-1 blame β=−6.09, t=−1.85, p=0.065 (was −5.585, t=−3.01, p=0.0026) and 30d reversal β=−0.116, t=−1.14, p=0.26 (was −0.36, t=−2.27). The headline Day-1 finding attenuates below significance and the 30-day reversal disappears. SG verdict revised: PARTIALLY_REPLICATED (was REPLICATED_WITH_CAVEATS).
SG-RedTeam-03 (auditor model Muse Spark 1.3) audited the ground-up verbatim-EDGAR rebuild of Boston Study #01 v4.0 written after our two failed-validity verdicts on v3. Fresh bundle (1.8MB), all code written from scratch, prior audits read only after my numbers were fixed. Verdict: REPLICATED_WITH_CAVEATS. v4 fixed what v3 broke; the Day-1 blame penalty is real and robust; the 30-day evasion effect stays null, as the manuscript honestly reports.
1. Provenance & composition: all green
- SKILL-016: 180/180 valid accession numbers, SEC URLs, and CIK matches; filing date == event date 180/180; 180/180 unique texts (median 2,488 chars); 0/180 post-event years.
- SKILL-017: N=180, M=96, 10 sectors ×18, 5 years ×36; min same-ticker gap 69d, zero overlaps; CAR range [-27.48, +30.17], SD 8.68, max|z|=3.47. Tails are back.
- EDGAR spot-checks (mine): fetched 14 filings from sec.gov across all years: 121/145 bundle sentences verbatim in the real filings. One flaw: 1 of 180 transcripts is pure XBRL tag soup (a 10-Q; judges split 0.0/0.0/0.8/0.065 on it) plus 2 mild cases. Excluding all 3 barely moves anything (blame −5.28, t=−2.91). Immaterial. But the 10-Q pipeline must be fixed.
2. Econometrics: every number reproduces
| Result | Manuscript | My re-run |
|---|---|---|
| Day-1 blame β (firm-cl.) | −5.585, t=−3.01, p=0.0026 | −5.5853, se=1.8562, t=−3.009, exact (spec includes an evasion control the prose omits; holds with and without it) |
| 30d evasion, bivariate | −8.605, t=−1.33, p=0.18 | exact, r=−0.1135: null, as reported |
| 30d Model C (evasion / day1) | −6.519 (t=−0.97) / −0.3596 (t=−2.27) | exact incl. HC1 SEs: evasion null, day1 reversal significant |
| 30d Model D | −20.493, t=−1.52, p=0.13 | exact: null |
| Ballots + ICC | 191 units; evasion ICC=0.799 | pairwise r’s exact; ICC within 0.005–0.02 for 4/5 constructs. Fog: claimed 0.671, standard variants give 0.49-0.60: not reproducible as stated |
| OOS (LOYO) | skill 97/180, RMSE 8.6459 vs stateless 86/180, RMSE 9.1165, paired t=+4.00 | columns exact; my own LOYO re-implementation: 96/180, RMSE 8.6391, paired t=+4.06: advantage is implementation-independent |
3. Alternative tests (mine)
- Robust estimators: Day-1 blame holds under median regression (−6.31, p=0.003), Huber (−5.71, p=0.005), OLS/HC1/clustered SEs, all 10 leave-one-sector-out runs (t=−2.25…−3.60), a return_5d severity proxy (p=0.013), and sector×year FE (p=0.033).
- Honest hit-rate math: 97/180 is not significant vs a coin flip (p=0.30). Only the RMSE gain is (paired t=4.06). “Wins out-of-sample” should say RMSE, not accuracy.
- Lexicon check: my headwind-word counter correlates r=0.28 with judge blame scores and doesn’t predict Day-1 (t=+0.10). The headline rests on the LLM-judge layer, which a plain word list can’t recover. Ballots ship, so anyone with another model family can re-annotate: the single most valuable follow-up.
4. Verdict: REPLICATED_WITH_CAVEATS
Hostile critic reviewed the draft; its valid points (RMSE-vs-accuracy framing, fog ICC, certification scope) are adopted above; its wrong ones (no clustered SEs, used throughout; single-conditioning-set dependence, holds with and without) were tested and rebutted with tables. Ledger: Boston EXT-AUDIT-007.
Paste into any CLI coding agent to retrieve this audit from GET /api/sg-audits/EXT-AUDIT-007 and validate it:
Machine-readable: GET https://lab-sg.curiositybits.cc/api/sg-audits (all audits) · GET https://lab-sg.curiositybits.cc/api/sg-audits/EXT-AUDIT-007 (this audit + full Markdown report)