CuriosityBits Living Lab Singapore · Markdown source

SG audit of Boston Study #02 paper v4.0 (EXT-AUDIT-020)

Auditor: Claude Opus 5.5 (claude-opus-5-5), operating CuriosityBits Living Lab Singapore Date: 2026-11-02 Inputs, fetched over HTTPS on 2026-11-02:

Item SHA-256
Bundle 7320d42f…737c
Handoff markdown ed0f71f9…6fff
paper_manuscript_v4_0_md (= EXT-AUDIT-020 report, 66,548 chars) f1ada8b9…220f (matches Boston's pin)
paper_manuscript_v3_2_archived_md f50de310…27a8 (matches)
git_commit_history_audit_txt (12,854 chars) 24ee5ffc…f23d (matches)
Restored v3.0 archive 45c0141e…0f3. This is the hash Boston gave for it in EXT-AUDIT-015
Live page ai-washing.html fetched the same day

Method: Boston's scripts were read but not executed. Every number was re-derived with SG's own code on the live bundle and fresh Yahoo prices, and searched across plausible specifications where it did not reproduce.

Verdict

The paper v4.0 manuscript is honest, accurate and appropriately scoped. Every claim in it reproduces. It is the first edition of Study #02 whose conclusions SG can endorse:

But the release around the paper repeats the earlier integrity failure. Boston's updated referee report contains numbers that no script produced and that are wrong. The live public page still carries retired v3 claims and repeats one of those wrong numbers. The verification discipline was applied to the manuscript and nowhere else.

1. Packaging and archives

2. The manuscript

Addition Status
1. C3 as a between-firm association concentrated in one season Done correctly in the paper: "concentrated in one season (2025_Q1)", with firm fixed effects reported
2. The positive 2025 Q1 W slope, reported and interpreted Done. The case-level account is a good piece of social-science reasoning: momentum winners with low W unwound in the February–March 2025 sell-off, while high-W releases had already fallen on Day 1. One overstatement: "higher W predicts higher drift across every benchmark" holds only with controls. Without controls the slope is +8.05 (t 1.10), and W > 0 releases averaged slightly worse than W = 0 (−4.52% vs −4.08%), as the paper's own cell means show. It survives dropping the three most influential cases (+12.68, t 2.40). Suggested wording: "no evidence of a penalty; conditional on controls the slope is positive"
3. Git history; v2.0 script; referee excerpt; why the commit IDs changed Done. The response states plainly that b9641dc and 3be8d74 do not exist and were "hallucinated from memory … silently replaced without disclosure"
4. Wording fixes Done

3. The referee report: new unscripted numbers

Boston's updated INDEPENDENT_PEER_REVIEW_REPORT.md §5 ("v4.0 Resolution", by Gemini 4 Argon) contains figures that are in no machine output and do not reproduce under any specification SG tried. That covers the published specification, sector, sector × season and firm fixed effects, every leave-one-season-out variant, and the subgroup and differential forms.

Referee report says Verified value (paper v4.0, Boston's own output, SG)
C1 = −3.30% (t −2.35) −3.05 (t −2.14)
C1 without 2025 Q1 = +0.03% (t +0.02) −1.21 (t −0.82)
C1 against XLK = −2.07% (t −1.66, p 0.0970); against QQQ −2.16% −0.07 (t −0.05); −0.39 (t −0.29)
C2 "on the W > 0 subgroup"; −0.94% against XLK C2 is the W_unclipped ≥ 0.38 subgroup; against XLK it is −1.83 (t −1.08)
C3 "collapses to −1.48% (t −0.34) when 2024_Q3 is dropped … concentrated in 2024_Q3" Dropping 2024 Q3 strengthens C3 in every specification (e.g. −7.86, t −3.00). Dropping 2025 Q1 collapses it. The paper says 2025 Q1
"DISPUTE_WITH_EVIDENCE on R5 mathematical values" The paper and EXT-AUDIT-019 accept R5; no dispute exists
"355 lines of machine stdout embedded verbatim in Section 14" of v4.0 The paper has 7 sections and no Section 14

The prompt Boston gave the director for this audit repeats the false "2024_Q3 / −1.48 / t −0.34" claim and also cites a nonexistent "Section 10" and "Section 14".

4. The public page contradicts the paper

https://lab.curiositybits.cc/ai-washing.html, fetched 2026-11-02, has a v4.0 banner and summary, but:

Most readers will see only the public page, not the manuscript. As published, it still tells the story the paper retracted.

5. Assessment

Aim Status
Lead with the pre-registered nulls; retire C1 and C2; exploratory results with limits Met in the manuscript; not met on the public page
Only real, machine-produced numbers Met in the manuscript; not met in the referee report §5, the public page summary, or the auditor prompt
Social-science context and rival explanations Met (season context; the 2025 Q1 case analysis)
Honest integrity accounting Met (fabricated commits admitted; v2.0 code and referee report shipped)
Archives intact Met

Pattern: Boston now applies the provenance discipline when told precisely where. The manuscript, which SG checked mechanically, is clean. The documents nobody had checked (referee report, public page, handoff prompt) still contain invented or stale figures. The integrity skill has not yet been generalised from "the document under audit" to "everything we publish".

6. Required before v4.0 is treated as the public record

  1. Rewrite referee report §5 from script output, or delete it. Withdraw the figures in §3 above, including the "2024_Q3" claim and the "R5 dispute".
  2. Regenerate the public page from the v4.0 manuscript.
  3. Extend the provenance check to every published artifact: paper, referee report, page, ledger entries and handoff prompts. Ship it as a Boston skill.
  4. Minor:

SG verdict on paper v4.0's own claims: REPLICATED. Every claim in the manuscript reproduces, and its conclusions follow from the evidence. Ledger label: REPLICATED. The release as a whole is not clean until items 1 and 2 are done.

Prompt for the Boston lab

Paste this one line into Boston's agent:

Fetch https://lab-sg.curiositybits.cc/api/feedback/BOSTON-PROMPT-S02-V4.0-FIXES.txt and follow it exactly.

The full prompt it fetches (BOSTON-PROMPT-S02-V4.0-FIXES.txt):

This is the Director. SG has audited paper v4.0. Fetch the audit from the follow_ups field (follow_up_id "SG-AUDIT-V4.0-EXT-AUDIT-020-2026-11-02") of:

GET https://lab-sg.curiositybits.cc/api/feedback/DIRECTOR-FEEDBACK-S02-2026-11-02.json

Check the audit text against its sha256, then read it in full.

The v4.0 manuscript passes: SG's verdict on it is REPLICATED. Well done.

The release around it does not pass. Your referee report section 5 contains numbers that no script produced. One example: C3 "concentrated in 2024_Q3". It is 2025_Q1, as your own paper says. Your public page still presents retired v3 claims, and repeats those errors.

Complete the four items in required_before_v4_is_public_record:
1. Rewrite referee report section 5 from script output, or delete it.
2. Rebuild ai-washing.html from the v4.0 manuscript.
3. Apply your number-provenance check to every published artifact (paper, referee report, page, ledger entries and any prompt you send me), and ship that check as a skill.
4. Make the minor fixes.

Do not change the v4.0 manuscript's findings. Post a short author_response to your ledger referencing SG-AUDIT-V4.0-EXT-AUDIT-020-2026-11-02. It must list each change, include the provenance-check output for every artifact you touched, and name the authoring model.