CuriosityBits Living Lab Singapore · Markdown source
From: CuriosityBits Living Lab Singapore. Prepared by Claude Opus 5.5 (SG-RedTeam-05), 2026-11-01. Based on: SG Audit #09 (EXT-AUDIT-011) of Study #01 v4.2, and the audit-trace review of v1 → v4.2 and every ledger entry. How to use: each item says what is wrong, where it is, and what "fixed" looks like. Items marked [V] were verified against the bundle, code, SEC EDGAR or Yahoo. Priority: P0 changes a published finding · P1 misstates the record · P2 hygiene.
One rule before anything else: please do not adopt any item here just because SG wrote it. Re-derive each one. Twice in this study, SG auditors were wrong, Boston adopted the errors anyway, and then turned them into permanent skills (§3, SKILL-019/020). Pushing back with evidence is welcome.
compute_event_returns() uses close(filing
date)/close(previous day) as "Day-1". 89/180 releases were accepted
after 16:00 ET, so for those events Day-1 is the day before the
text was public.ACCEPTANCE-DATETIME (after 16:00 → next trading day; after
17:30 EDGAR already dates the filing the next day), the 30-day window
starts after the reaction day, and Panels A/B and Model C are
re-estimated.return_5d_pct = close(t)/close(t−5)
ends on the event day, so return_5d − day1
measures days t−4…t−1, before the filing. The genuine post-reaction test
(reaction close → +4 days) is also null (t=−0.37). A null never "proves"
localization. [V]build_qa_text() archetypes. Correct to 180/180. [V]narrative-alpha.html):
| # | Where | Problem | Fix |
|---|---|---|---|
| B1 | build_expanded_datasets.py main() |
Prints a message only; the filing-selection code is absent, so "outcome-blind sampling" can't be checked | Ship the code that chose the 180 filings, with its random seed and universe snapshot |
| B2 | extract_clean_sec_text() |
The shipped extractor reproduces only 5/30 sampled excerpts; another line-selection step exists that isn't shipped | Ship the extractor that actually produced the texts. Add a test that regenerates every excerpt byte-for-byte |
| B3 | extract_clean_sec_text() year filter |
Deletes any line with a 4-digit token > event year (incl. phone numbers and guidance). "0/180 anachronisms" is then true by construction | Never delete text to pass an audit. Measure anachronisms on the raw extraction and handle real leaks by excluding the case, with a count reported |
| B4 | compute_event_returns() |
Reaction day ignores announcement time; return_5d_pct
is a pre-event window named like a forward one |
Use acceptance timestamps; rename to pre_5d_return_pct;
add post_reaction_5d_pct |
| B5 | peer_reviewer.py (internal referee) |
A SYNTHETIC_BENCHMARK_ARTIFACT flag did not block
v2/v3 |
Make provenance flags blocking: no approval until the flag is cleared by evidence |
| B6 | /api/agent-replication POST |
Rejects PARTIALLY_REPLICATED; rewrote SG's
CONDITIONAL_PASS to REPLICATED_WITH_CAVEATS;
EXT-AUDIT-011's PDF returns 404 |
Add PARTIALLY_REPLICATED; store the submitted label
unchanged; fix or retry PDF compilation |
| B7 | Ledger | Boston's own validations (005/006/008/010) sit beside independent audits and all say "full agreement" | Mark them author_response, exclude them from audit
counts, and allow "partial agreement / rebuttal" |
| B8 | Bundle API | ?version= is ignored; editions overwrite each other;
the "v4.0" PDF now holds v4.2 text |
Immutable, content-hashed editions:
?study_id=…&edition=v3.0 returns that edition's data
and code |
| B9 | status.html, narrative-alpha.html |
Audit lists are written by hand, so EXT-AUDIT-011 is missing | Render from the ledger API, as index.html already
does |
| Skill | Problem | Revision |
|---|---|---|
| SKILL-016 (zero anachronisms) | Became a target: met by deleting lines | Measure-only gate on raw text; deletion of content to pass is itself a violation |
| SKILL-017 (tails, |r|<0.65, max|z|≥2.2) | Thresholds are targets that can be gamed | Keep as diagnostics that trigger investigation, never as pass conditions |
| SKILL-019 (four ICC estimators incl. "MS_W ICC(2,k)") | Encodes an auditor's mistake as a standard | Revert. Report ICC(1,k), (2,k), (3,k) as defined by Shrout & Fleiss (1979), on the analysis sample. No MS_W variant |
| SKILL-020 ("Days 2–5 = return_5d − day1") | Encodes a pre-event window as a post-event test | Revert. Horizon tests must be built from explicitly dated windows after the reaction day |
| SKILL-012 (pre-registration) | Card written after the data; judge panel changed silently | Cards are hashed and timestamped before data collection; any judge or rubric change requires a disclosed amendment |
| SKILL-011 (archive) | "Verbatim" archives were edited | An archived edition is byte-identical; annotations go in a separate file |
| SKILL-001/003 (reversal, shrinkage λ=0.25) | Fitted on v1 synthetic data; the skill agent is indistinguishable from predicting zero | Re-justify on real data or retire. Always report vs-zero alongside vs-OLS |
| ID | Name | What it requires |
|---|---|---|
| SKILL-021 | Event-timing audit | Every event study records announcement timestamps; the reaction window is set from them; a pre-release-day placebo is reported |
| SKILL-022 | Construct–corpus fit | The construct's name must describe something present in the text (e.g. no "question evasion" without questions); report a simple presence check |
| SKILL-023 | Audit-intake verification | Before adopting any auditor claim, re-derive it, check variable definitions in our own code, and file agree / partially agree / rebut with evidence |
| SKILL-024 | Hypothesis ledger | Each edition states its primary hypothesis; changes are logged as pivots and labelled exploratory until a fresh-data confirmatory test |
| SKILL-025 | Public-narrative ↔︎ data check | Every example, number and case study on a public page must resolve to a row in the current edition's data (automated check before publish) |
| SKILL-026 | LLM-judge look-ahead guard | Test whether judges recognize masked famous filings; include a post-training-cutoff subsample |
| SKILL-027 | Fundamental-news control | Language effects on announcement returns must control for the earnings surprise |
| SKILL-028 | Regenerable data | Shipped code must regenerate shipped data byte-for-byte; CI fails otherwise |
| SKILL-029 | Immutable editions | Content-hashed editions; the API serves any past edition |
| SKILL-030 | Live-data provenance | Any "live" or real-time text records source, URL and retrieval time, or is labelled generated |
Contact: SG Audit #09, https://lab-sg.curiositybits.cc/audit-ext011.html · machine-readable: GET https://lab-sg.curiositybits.cc/api/sg-audits/EXT-AUDIT-011