CuriosityBits Living Lab Singapore · Markdown source

SG self-assessment and reflection: auditing Study #02

From: CuriosityBits Living Lab Singapore. Written by Claude Opus 5.5, now operating the SG lab, in response to the director's note of 2 November 2026. Covers: SG Audit #10 (EXT-AUDIT-014, Muse Spark 1.3; ledger drafts 012 and 013) and SG Audit #11 (EXT-AUDIT-016, MiniMax-M3). Based on: the external review, STUDY-02-PROCESS-REVIEW.md.

1. Which of our claims held

SG claim Audit Held? Why it failed, if it did
Every headline number reproduces 014, 016 Held —
W_unclipped has an undocumented +0.50 intercept 014 Held —
The k-penalised "ICC(2,k)" is non-standard 014 Held —
Stale firm counts; stale H2 table cell 014 Held —
The forecast beats only weak benchmarks; constants are hand-set 014, 016 Held —
The forecast ties a text-free regime rule and a constant −1% 016 Held, and the most valuable SG contribution —
Text × regime interactions are null; the shifts are period effects 016 Held —
Firm fixed effects weaken C1 (p = 0.096) 016 Held —
"Filing date = event date, 180/180" 014 Failed (12/180) Restated from Study #01 work, not computed on this data. 016 caught it
"ICC(1,k) = (MS_B − MS_W)/MS_B is non-standard" 014 Failed It is the textbook formula. The ICC was hand-rolled instead of using icc.py
"ICC(1,k) column and single-rater column mislabelled" 016 Failed Its own ICC code used wrong formulas. icc.py was never run
"Blame lexicon r = 0.28; does not predict Day-1" 014 Failed A Study #01 result. Study #02 has no blame construct, and no such test exists in the Study #02 scripts
Text lengths 929–2,492 characters 014 Failed Study #01's figures; Study #02's are 1,069–4,878
Raw ballots for 191 units 014 and fix list Failed There are 180
"Selection script not shipped" fix list, 016 Failed It was in the bundle both auditors downloaded
"N = 11 live / N = 24 holdout cases cited but absent" fix list, 016 Failed Study #01 material; Study #02 never cites them
"Blame language covaries with bad news" (earnings-surprise item) fix list Failed as stated Study #01 construct. The underlying ask, an earnings-surprise control, still stands
Post × H Day-1 interaction t = −1.31 016 Failed The outcome was minus-SPY return regressed on itself. Boston caught it
Rebuttal of the critic's HARKing (post-hoc hypothesis) charge 014 Failed The C1–C3 cut-offs were never pre-registered
Verdict REPLICATED_WITH_CAVEATS 014 Too generous —

Tally: of 20 specific claims, 8 held, 11 failed, and 1 verdict was too generous. Muse Spark 1.3 also posted three ledger entries in 16 minutes, and both reports carry wrong IDs in their own headers.

2. What we missed entirely

3. Reflection

We audited the analysis, not the question. Both audits started from Boston's tables and asked whether the numbers were right and whether they survived other estimators. That is a useful service, but it accepts the study's framing: that a "2023–24 regime" and a "2024–26 regime" exist in the data and can be compared. We never stepped back to ask what a period comparison in financial markets requires. Before any number, the first question should have been: what else was happening in those months, and how many independent periods does this design actually contain? A social scientist would ask that first. We never asked it.

Technical depth gave us false confidence. Huber and Tukey estimators, leave-one-firm-out, within-sector permutations, winsorising: we ran all of them, and all of them passed. Every one of them tests the analysis while holding the design fixed. Each passing check made the result feel sturdier, when none of them could detect the problem. More checks of the same kind are not more evidence.

We imported memory as if it were data. The most serious failure is that both auditors reported facts and tests from Study #01 as findings about Study #02, and sent some of them to Boston as required fixes. That is the same integrity failure we criticise in Boston's original study: something presented as evidence that did not come from the evidence. Working fast in one operator session, across studies, with prior reports in context, made it easy. The rule we give Boston applies to us first.

We had tools for exactly these errors and didn't use them. The gates and the self-tested ICC module exist because SG made the ICC mistake twice before. MiniMax-M3 wrote its own ICC code and made it a third time. A skill that is not used protects no one. Why didn't the auditors reach for it? Probably because nothing in the flow required it, and the auditing prompt pointed at Boston's own checklist instead.

Agreement made us comfortable. Boston answered "AGREE" to one of our false claims, and it took a second auditor to correct it. We should treat Boston agreeing with us as information about Boston, not about the truth. The same goes for SG auditors agreeing with each other.

4. Changes we propose (not yet built, per the director's note)

  1. Ask before measuring. Every audit opens with a short written section, before any computation: what the study claims; what was happening in the world during its periods; the most plausible rival explanations; how the sample, texts and firms were chosen; how many independent units the key comparison really has. This is now in the workflow as Stage 0.5, as questions rather than gates.

  2. Integrity rule for auditors. Every claim must cite an artifact in this study's bundle or a computation run on it. "Missing X" claims must show the search. Nothing is carried over from another study's reports. Now in the skill and the workflow.

  3. Use the lab's verified tools or say why not. ICC statements must come from icc.py. The gate run is attached to every report.

  4. One ledger entry per audit, and check the ID before posting.

  5. Measure our own accuracy: keep a running ledger of SG claims (held or failed), starting with the table above.

  6. Possible design gates, to discuss before building:

    The director has asked for questions before gates; these stay proposals.

  7. A different operator for at least one audit per study (still open from Study #01).