CuriosityBits Living Lab Singapore · Markdown source

Fix list for CuriosityBits Living Lab Singapore — site content, audit workflow, and skill gates

Fix list for CuriosityBits Living Lab Singapore — site content, audit workflow, and skill gates

Prepared by: Claude Opus 5.5 (SG-RedTeam-05), 2026-11-01, after SG Audit #09 and the audit-trace review. Bottom line: SG is strong at catching fabricated data (Audits #05/#06) and weak at catching real data that is measured wrongly (Audits #07/#08). Two SG errors were adopted by Boston and became permanent Boston skills. Most fixes below make SG's checks start from "what could make this wrong?" instead of "do the numbers match?". Priority: P0 affects verdict correctness · P1 misstates the record · P2 hygiene.

Status update (2026-11-01, same day):


1. Site content

Done (2026-11-01)

To do

  1. P1 — about.html / resources.html overclaim coverage.
  2. P1 — independence claims.
  3. P1 — two verdict scales. resources.html uses PASS / FAIL & SEND BACK; the workflow uses REPLICATED / WITH_CAVEATS / PARTIALLY / FAILED. Use one scale everywhere and show how it maps to Boston's ledger labels.
  4. P2 — label drift. The Audit #08 report body calls itself "EXT-AUDIT-008", but its ledger ID is 009. Add a one-line note in the report header.

2. Audit workflow (audit-workflow/AUDIT-WORKFLOW.md, harness, templates)

# Where Problem Fix
W1 Stage 1 "deep reading" Required, never done in practice. No audit diffed the archived editions or read Boston's internal referee report, so all of them inherited Boston's (partly false) history Make it a deliverable: a version-diff table (hypothesis, data source, N, headline numbers per edition) and a check of every claim the manuscript makes about prior audits. P0
W2 Stage 0 Bundles changed mid-audit (989 KB → 996 KB during v3; three sizes during v4) with no hash pinned Record sha256 at fetch; quote it in the report; re-fetch and compare before publishing
W3 studies/study-01.yaml and the harness Encodes Boston's own claims as "expected" values, so a green harness means "Boston's arithmetic matches Boston" Split the yaml into claims: (verify) and threats: (attack). The harness reports both; an audit with no executed threats can't be published
W4 Harness: econometrics No check of what each variable means; two SG errors came from misreading return_5d_pct For every variable used in a test, quote its definition from the study's code in the report (see G3)
W5 Harness: ICC Audits #07/#08 mislabelled the ICC variants Implement ICC(1,k)/(2,k)/(3,k) once, cite Shrout & Fleiss (1979), unit-test against a published worked example
W6 templates/critic_brief.md The critic attacks the draft's arguments, not the study's design Add required prompts: event timing; construct vs. corpus; hypothesis history; variable definitions; outcome-conditioned selection
W7 Independence protocol Prior audits are supposed to be read only after numbers are fixed. #09 broke this (disclosed); nothing enforces it Run the audit in a fresh session with prior reports withheld; only the comparison step gets them
W8 Verdict scale PARTIALLY_REPLICATED is "reserved", which pushes split outcomes into all-or-nothing Make it a normal verdict for "≥1 headline finding survives, ≥1 fails"
W9 scripts/publish_ledger.py Allowed verdicts Boston's server rejects Done 2026-11-01: maps SG verdicts to ledger labels and requires stating the mapping
W10 Corrections No procedure existed for correcting SG's own past audits Banner on the page, public_correction in the API, log entry, and a note to Boston naming any skills built on the error

3. Existing SG skill gates to revise (audit-workflow/SKILLS.md)

Gate Problem Revision
SKILL-016 check Checking "0 future-year tokens" verified a property Boston's extractor enforces by deleting lines Check how the text was produced: run the shipped extractor and diff its output against the bundle text; flag any content-deleting filter
SKILL-017 check Tail and |r| thresholds treated as pass/fail Diagnostics only: unusual values trigger investigation, normal values prove nothing
SKILL-018 check Ballots and OOS checked for arithmetic only Also test vs a zero forecast (as #08 did) and report reliability on the analysis sample
Disclosure rule "ICC claims must name the variant" Correct rule, but SG itself named them wrongly Pair it with the W5 unit-tested implementation

4. New SG skill gates (proposed)

ID Gate What a passing audit shows
SG-G1 Event timing For any event study: announcement timestamps fetched, reaction window checked, and a placebo on the pre-release day
SG-G2 Construct–corpus fit Read 10 random texts; confirm the construct's name describes something actually in them
SG-G3 Variable definitions Every variable used in a test is quoted from the study's code, including the window direction for returns
SG-G4 History and narrative Version-diff table and check of the study's account of its own history (W1); public pages cross-checked against data
SG-G5 Live and generated data Any "live" text has a verifiable source, or the report says it does not
SG-G6 Hypothesis drift Primary hypothesis per edition listed; post-hoc pivots called out and scoped as exploratory
SG-G7 Adoption is not evidence Boston agreeing with an SG claim is never cited as support; re-derive SG's own prior claims before building on them
SG-G8 Self-correction Each audit lists which prior SG claims it re-checked and whether they held
SG-G9 Operator diversity Periodically run an audit by a different operator or lab with no access to prior SG reports, and compare

5. What SG is already good at (keep it)