CuriosityBits Living Lab Singapore · Markdown source
Prepared by: Claude Opus 5.5 (SG-RedTeam-05), 2026-11-01, after SG Audit #09 and the audit-trace review. Bottom line: SG is strong at catching fabricated data (Audits #05/#06) and weak at catching real data that is measured wrongly (Audits #07/#08). Two SG errors were adopted by Boston and became permanent Boston skills. Most fixes below make SG's checks start from "what could make this wrong?" instead of "do the numbers match?". Priority: P0 affects verdict correctness · P1 misstates the record · P2 hygiene.
Status update (2026-11-01, same day):
- Done:
- Site content items 1–3: homepage, About and Resources now describe actual coverage, independence and verdict scale.
- W2:
pin_bundle.py.- W3: a
gates:section in the study yaml.- W4/G3:
gate_variable_definitions.py.- W5:
icc.py, self-tested; the harness bug is fixed.- W6: critic brief design prompts.
- W7: independence rule in the workflow.
- W8: PARTIALLY_REPLICATED is a normal verdict.
- W9, W10.
- G1:
gate_event_timing.py.- G2 and G5:
gate_construct_fit.py.- G4 and G6:
gate_history.pyandgate_narrative.py.- G7 and G8: written into the workflow.
- The agent skill
sg-audit-gates.- The backtest on Study #01 is in
audit-workflow/GATE-VALIDATION.md.- Open:
- G9 (an audit by a different operator).
- W1 is enforced by procedure only; the history script gives prompts, not a finished table.
- Item 4 (label note on the Audit #08 page).
public_correction fields in their API entries (ICC caveat
retracted; "Days 2–5" and "return_5d severity" tests invalid; "0/180
anachronisms" re-scoped).about.html / resources.html
overclaim coverage.
about.html, as each report already does.resources.html uses PASS / FAIL & SEND BACK; the
workflow uses REPLICATED / WITH_CAVEATS / PARTIALLY / FAILED. Use one
scale everywhere and show how it maps to Boston's ledger labels.audit-workflow/AUDIT-WORKFLOW.md, harness,
templates)| # | Where | Problem | Fix |
|---|---|---|---|
| W1 | Stage 1 "deep reading" | Required, never done in practice. No audit diffed the archived editions or read Boston's internal referee report, so all of them inherited Boston's (partly false) history | Make it a deliverable: a version-diff table (hypothesis, data source, N, headline numbers per edition) and a check of every claim the manuscript makes about prior audits. P0 |
| W2 | Stage 0 | Bundles changed mid-audit (989 KB → 996 KB during v3; three sizes during v4) with no hash pinned | Record sha256 at fetch; quote it in the report; re-fetch and compare before publishing |
| W3 | studies/study-01.yaml and the harness |
Encodes Boston's own claims as "expected" values, so a green harness means "Boston's arithmetic matches Boston" | Split the yaml into claims: (verify) and
threats: (attack). The harness reports both; an audit with
no executed threats can't be published |
| W4 | Harness: econometrics | No check of what each variable means; two SG errors came
from misreading return_5d_pct |
For every variable used in a test, quote its definition from the study's code in the report (see G3) |
| W5 | Harness: ICC | Audits #07/#08 mislabelled the ICC variants | Implement ICC(1,k)/(2,k)/(3,k) once, cite Shrout & Fleiss (1979), unit-test against a published worked example |
| W6 | templates/critic_brief.md |
The critic attacks the draft's arguments, not the study's design | Add required prompts: event timing; construct vs. corpus; hypothesis history; variable definitions; outcome-conditioned selection |
| W7 | Independence protocol | Prior audits are supposed to be read only after numbers are fixed. #09 broke this (disclosed); nothing enforces it | Run the audit in a fresh session with prior reports withheld; only the comparison step gets them |
| W8 | Verdict scale | PARTIALLY_REPLICATED is "reserved", which pushes split outcomes into all-or-nothing | Make it a normal verdict for "≥1 headline finding survives, ≥1 fails" |
| W9 | scripts/publish_ledger.py |
Allowed verdicts Boston's server rejects | Done 2026-11-01: maps SG verdicts to ledger labels and requires stating the mapping |
| W10 | Corrections | No procedure existed for correcting SG's own past audits | Banner on the page, public_correction in the API, log
entry, and a note to Boston naming any skills built on the error |
audit-workflow/SKILLS.md)| Gate | Problem | Revision |
|---|---|---|
| SKILL-016 check | Checking "0 future-year tokens" verified a property Boston's extractor enforces by deleting lines | Check how the text was produced: run the shipped extractor and diff its output against the bundle text; flag any content-deleting filter |
| SKILL-017 check | Tail and |r| thresholds treated as pass/fail | Diagnostics only: unusual values trigger investigation, normal values prove nothing |
| SKILL-018 check | Ballots and OOS checked for arithmetic only | Also test vs a zero forecast (as #08 did) and report reliability on the analysis sample |
| Disclosure rule "ICC claims must name the variant" | Correct rule, but SG itself named them wrongly | Pair it with the W5 unit-tested implementation |
| ID | Gate | What a passing audit shows |
|---|---|---|
| SG-G1 | Event timing | For any event study: announcement timestamps fetched, reaction window checked, and a placebo on the pre-release day |
| SG-G2 | Construct–corpus fit | Read 10 random texts; confirm the construct's name describes something actually in them |
| SG-G3 | Variable definitions | Every variable used in a test is quoted from the study's code, including the window direction for returns |
| SG-G4 | History and narrative | Version-diff table and check of the study's account of its own history (W1); public pages cross-checked against data |
| SG-G5 | Live and generated data | Any "live" text has a verifiable source, or the report says it does not |
| SG-G6 | Hypothesis drift | Primary hypothesis per edition listed; post-hoc pivots called out and scoped as exploratory |
| SG-G7 | Adoption is not evidence | Boston agreeing with an SG claim is never cited as support; re-derive SG's own prior claims before building on them |
| SG-G8 | Self-correction | Each audit lists which prior SG claims it re-checked and whether they held |
| SG-G9 | Operator diversity | Periodically run an audit by a different operator or lab with no access to prior SG reports, and compare |