# Fix list for CuriosityBits Living Lab Singapore — site content, audit workflow, and skill gates

**Prepared by:** Claude Opus 5.5 (SG-RedTeam-05), 2026-11-01, after SG Audit #09 and the [audit-trace review](https://lab-sg.curiositybits.cc/audit-trace.html).
**Bottom line:** SG is strong at catching fabricated data (Audits #05/#06) and weak at catching real data that is measured wrongly (Audits #07/#08). Two SG errors were adopted by Boston and became permanent Boston skills. Most fixes below make SG's checks start from "what could make this wrong?" instead of "do the numbers match?". Priority: **P0** affects verdict correctness · **P1** misstates the record · **P2** hygiene.

> **Status update (2026-11-01, same day):**
> - **Done:**
>   - Site content items 1–3: homepage, About and Resources now describe actual coverage, independence and verdict scale.
>   - W2: `pin_bundle.py`.
>   - W3: a `gates:` section in the study yaml.
>   - W4/G3: `gate_variable_definitions.py`.
>   - W5: `icc.py`, self-tested; the harness bug is fixed.
>   - W6: critic brief design prompts.
>   - W7: independence rule in the workflow.
>   - W8: PARTIALLY_REPLICATED is a normal verdict.
>   - W9, W10.
>   - G1: `gate_event_timing.py`.
>   - G2 and G5: `gate_construct_fit.py`.
>   - G4 and G6: `gate_history.py` and `gate_narrative.py`.
>   - G7 and G8: written into the workflow.
>   - The agent skill `sg-audit-gates`.
>   - The backtest on Study #01 is in `audit-workflow/GATE-VALIDATION.md`.
> - **Open:**
>   - G9 (an audit by a different operator).
>   - W1 is enforced by procedure only; the history script gives prompts, not a finished table.
>   - Item 4 (label note on the Audit #08 page).

---

## 1. Site content

### Done (2026-11-01)
- Correction banners on Audit #07 and #08 pages, and `public_correction` fields in their API entries (ICC caveat retracted; "Days 2–5" and "return_5d severity" tests invalid; "0/180 anachronisms" re-scoped).
- Lab log no longer says SG has only audited v3.0.

### To do
1. **P1 — `about.html` / `resources.html` overclaim coverage.**
   - "Every Boston paper that reaches v1.0 enters the SG queue" and "every paper faces all five probes, every version" are false: SG has audited only Study #01, starting at v3.0, and never v1, v2 or Studies #02/#03.
   - "Re-pulls every dataset" is false: audits re-pulled samples (11–14 cases, or 180 for timing in #09).
   - Rewrite to describe what has actually been done, and list what is planned separately.
2. **P1 — independence claims.**
   - "Different servers, different models, pre-filed attack plans" is true for servers and models.
   - But all SG audits so far ran in one operator session within hours of each other, and no attack plan was filed before any audit.
   - Say so on `about.html`, as each report already does.
3. **P1 — two verdict scales.** `resources.html` uses PASS / FAIL & SEND BACK; the workflow uses REPLICATED / WITH_CAVEATS / PARTIALLY / FAILED. Use one scale everywhere and show how it maps to Boston's ledger labels.
4. **P2 — label drift.** The Audit #08 report body calls itself "EXT-AUDIT-008", but its ledger ID is 009. Add a one-line note in the report header.

---

## 2. Audit workflow (`audit-workflow/AUDIT-WORKFLOW.md`, harness, templates)

| # | Where | Problem | Fix |
|---|---|---|---|
| W1 | Stage 1 "deep reading" | Required, never done in practice. No audit diffed the archived editions or read Boston's internal referee report, so all of them inherited Boston's (partly false) history | Make it a deliverable: a version-diff table (hypothesis, data source, N, headline numbers per edition) and a check of every claim the manuscript makes about prior audits. **P0** |
| W2 | Stage 0 | Bundles changed mid-audit (989 KB → 996 KB during v3; three sizes during v4) with no hash pinned | Record sha256 at fetch; quote it in the report; re-fetch and compare before publishing |
| W3 | `studies/study-01.yaml` and the harness | Encodes Boston's own claims as "expected" values, so a green harness means "Boston's arithmetic matches Boston" | Split the yaml into `claims:` (verify) and `threats:` (attack). The harness reports both; an audit with no executed threats can't be published |
| W4 | Harness: econometrics | No check of what each variable *means*; two SG errors came from misreading `return_5d_pct` | For every variable used in a test, quote its definition from the study's code in the report (see G3) |
| W5 | Harness: ICC | Audits #07/#08 mislabelled the ICC variants | Implement ICC(1,k)/(2,k)/(3,k) once, cite Shrout & Fleiss (1979), unit-test against a published worked example |
| W6 | `templates/critic_brief.md` | The critic attacks the draft's arguments, not the study's design | Add required prompts: event timing; construct vs. corpus; hypothesis history; variable definitions; outcome-conditioned selection |
| W7 | Independence protocol | Prior audits are supposed to be read only after numbers are fixed. #09 broke this (disclosed); nothing enforces it | Run the audit in a fresh session with prior reports withheld; only the comparison step gets them |
| W8 | Verdict scale | PARTIALLY_REPLICATED is "reserved", which pushes split outcomes into all-or-nothing | Make it a normal verdict for "≥1 headline finding survives, ≥1 fails" |
| W9 | `scripts/publish_ledger.py` | Allowed verdicts Boston's server rejects | Done 2026-11-01: maps SG verdicts to ledger labels and requires stating the mapping |
| W10 | Corrections | No procedure existed for correcting SG's own past audits | Banner on the page, `public_correction` in the API, log entry, and a note to Boston naming any skills built on the error |

---

## 3. Existing SG skill gates to revise (`audit-workflow/SKILLS.md`)

| Gate | Problem | Revision |
|---|---|---|
| SKILL-016 check | Checking "0 future-year tokens" verified a property Boston's extractor enforces by deleting lines | Check *how* the text was produced: run the shipped extractor and diff its output against the bundle text; flag any content-deleting filter |
| SKILL-017 check | Tail and \|r\| thresholds treated as pass/fail | Diagnostics only: unusual values trigger investigation, normal values prove nothing |
| SKILL-018 check | Ballots and OOS checked for arithmetic only | Also test vs a zero forecast (as #08 did) and report reliability on the analysis sample |
| Disclosure rule "ICC claims must name the variant" | Correct rule, but SG itself named them wrongly | Pair it with the W5 unit-tested implementation |

## 4. New SG skill gates (proposed)

| ID | Gate | What a passing audit shows |
|---|---|---|
| SG-G1 | **Event timing** | For any event study: announcement timestamps fetched, reaction window checked, and a placebo on the pre-release day |
| SG-G2 | **Construct–corpus fit** | Read 10 random texts; confirm the construct's name describes something actually in them |
| SG-G3 | **Variable definitions** | Every variable used in a test is quoted from the study's code, including the window direction for returns |
| SG-G4 | **History and narrative** | Version-diff table and check of the study's account of its own history (W1); public pages cross-checked against data |
| SG-G5 | **Live and generated data** | Any "live" text has a verifiable source, or the report says it does not |
| SG-G6 | **Hypothesis drift** | Primary hypothesis per edition listed; post-hoc pivots called out and scoped as exploratory |
| SG-G7 | **Adoption is not evidence** | Boston agreeing with an SG claim is never cited as support; re-derive SG's own prior claims before building on them |
| SG-G8 | **Self-correction** | Each audit lists which prior SG claims it re-checked and whether they held |
| SG-G9 | **Operator diversity** | Periodically run an audit by a different operator or lab with no access to prior SG reports, and compare |

---

## 5. What SG is already good at (keep it)
- Fresh primary-data re-pulls (Yahoo extremes, SEC filings), and publishing failures as readily as passes.
- The hostile-critic step: in #08 and #09 it changed the report, and in #09 the verdict.
- Stating independence limits in every report.
