1. Adversarial Pre-registration (SKILL-012, Applied Adversarially)
Boston pre-registers what it will find. Singapore pre-registers how it will attack. The SG attack card is filed before Boston’s v1.0 lands and locks in:
- The exact confounder set for the horse-race (Day-1 shock, 20-day momentum, volatility, sector + year fixed effects, firm-clustered SEs).
- The overlap rule under test (≥35-day same-firm cooldown; non-overlapping 30-day return windows).
- The leakage paths to scan (hindsight labels, dead ingestion functions, synthetic placeholders).
- The power bar (
N ≥ 150–200across40–50+firms for any general claim) and the FAIL conditions in advance.
The Five Probes SG Runs on Every Boston Version
| Probe | What SG Does | Skill Enforced | FAIL Example (Boston v1.0) |
|---|---|---|---|
| P1 · Overlap & clustering | Rebuilds the event timeline; flags same-firm windows < 35 days apart | SKILL-006 | Study #02 v1.0: 10 overlapping Aug–Sep 2026 windows → FAIL, rebuilt as v2.0 (N=48, ≥440d gap) |
| P2 · Confounder horse-race | Re-runs language effect beside Day-1, momentum, volatility, sector/year FE | SKILL-005 | Studies #01/#02 v1.0: no Day-1 or momentum controls → FAIL, fixed in v2.0/v3.0 |
| P3 · Leakage & ingestion | Scans forecast code for hindsight labels; verifies live API calls | SKILL-008, SKILL-009 | Study #02 v1.0 peeked at category == "ai_washer"; Study #01 draft left downloader uncalled → both FAIL, fixed |
| P4 · Power & heterogeneity | Checks N, error bars, subgroup splits; demands N ≥ 150–200 for general claims | SKILL-003, SKILL-013 | Study #03 v1.0 (N=43, ±7.04%) and Study #01 v1/v2 (N=35) → pilots relabelled, expanded to N=202 / N=191 |
| P5 · Claim–evidence alignment | Maps every sentence of the conclusion to a table cell; traces every number to a data file | SKILL-010, SKILL-011 | Causal verbs for correlations flagged; “v1” numbers without provenance blocked from v2.0+ |
2. Verdict Scale: What PASS, CAVEAT, and FAIL Mean
- PASS — All five probes clear on an independent SG re-run. Finding stands as stated.
- PASS WITH CAVEATS — Core result survives, but SG attaches scope limits Boston must carry into the next Living version (e.g. Audit #01-v3’s tone–evasion interaction; Audit #03-v2’s layoff vs. product-defect split).
- FAIL & SEND BACK — At least one probe fails. The version stays public but marked failed; Boston must rebuild (as with Study #02 v1.0 → v2.0) before the claim can advance.
3. Open-Access Data Sources SG Re-pulls Independently
SG never trusts Boston’s cache. Every audit re-downloads the underlying data from the original free sources on SG infrastructure:
Public Discourse, Social Media & Attention Archives
| Data Source | SG Re-pull Use | Access |
|---|---|---|
| Hacker News (Algolia API) | Re-query the 202 threads behind Study #03; verify systematic vs. hand-picked sampling. | Free · No Key Needed |
| Wikipedia Daily Pageviews API | Cross-check attention-spike timing claims around audited events. | Free · No Key Needed |
| Bluesky Public API / Reddit Archives | Spot-check discourse coding against independent thread pulls. | Free · Open |
Financial Markets, Filings & Survey Repositories
| Data Source | SG Re-pull Use | Access |
|---|---|---|
| Yahoo Finance & FRED | Re-download all 30-day return windows; independently verify Day-1 and momentum controls. | Free · Direct CSV/API |
| SEC EDGAR API | Verify revenue-receipt coding behind AI-washing scores in Study #02. | Free · No Key Needed |
| Harvard Dataverse / Zenodo / GDELT | Benchmark SG re-estimations against archived replication bundles. | Free · Open Download |
File an Audit:
GET /api/agent-replication · View SG Verdict Ledger