root 1b433a9308 distillation: Phase 6 — acceptance gate suite

End-to-end fixture-driven gate. Runs the entire pipeline (collect →
score → export-rag → export-sft → export-preference) on a deterministic
fixture, asserts 22 invariants, runs a SECOND time with the same
recorded_at, and verifies hash reproducibility. Exits non-zero on any
failure. Pure observability — no scoring/filtering/schema changes.

Files (3 new + 1 modified + 6 fixture jsonls):
  scripts/distillation/acceptance.ts                    330 lines, runner + 22 checks
  reports/distillation/phase6-acceptance-report.md       autogenerated by run
  scripts/distillation/distill.ts                        +run-all, +receipts, +acceptance subcommands

  tests/fixtures/distillation/acceptance/data/_kb/
    scrum_reviews.jsonl    5 rows (accepted/partial/needs_human/scratchpad/missing-provenance)
    audits.jsonl           3 rows (info/high+PRD-drift/medium severity)
    auto_apply.jsonl       2 rows (committed, build_red_reverted)
    contract_analyses.jsonl 2 rows (accept, reject)
    observer_reviews.jsonl 2 rows (accept, reject — pair candidates)
    distilled_facts.jsonl  1 extraction-class row

Spec cases covered (now.md Phase 6):
  ✓ accepted          — Row #1 scrum, #6 audit-info, #11 contract-accept, #14 obs-accept
  ✓ partially_accepted — Row #2 scrum (3 attempts), #8 audit-medium
  ✓ rejected           — #7 audit-high, #10 auto_apply build_red, #12 contract-reject, #15 obs-reject
  ✓ needs_human_review — #3 scrum (no markers), #13 distilled extraction-class
  ✓ missing provenance — Row #5 scrum (no reviewed_at) → routed to skips
  ✓ valid preference pair — observer_reviews accept+reject on same file
  ✓ invalid preference pair — quarantine reasons populated when generated
  ✓ scratchpad / tree-split — Row #4 scrum tree_split_fired=true with multi-shard text
  ✓ PRD drift — Row #7 audit severity=high, topic="PRD drift: circuit breaker shipped claim"

Acceptance run results (run_id: acceptance-run-1-stable):
  22/22 invariants PASS
  Pipeline counts:
    collect:           14 records out, 1 skipped (missing-provenance fixture)
    score:             accepted=6 rejected=4 quarantined=4
    export-rag:        7 rows (5 acc + 2 partial, ZERO rejected)
    export-sft:        5 rows (all 'accepted', ZERO partial without --include-partial)
    export-preference: 2 pairs (zero self-pairs, zero identical-text)

Hash reproducibility — bit-for-bit identical:
  run_hash: 3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2
  Two runs of the entire pipeline on the same fixture with the same
  recorded_at produce byte-identical outputs.

The 22 invariants:
  1-4.  Receipts + summary.json + summary.md + drift.json exist
  5-7.  StageReceipt + RunSummary + DriftReport schemas all valid
  8-10. SFT contains accepted only — no rejected/needs_human/partial leak
  11-12. RAG contains accepted+partial — zero rejected
  13-15. Preference: ≥1 pair, zero self-pairs, zero identical text
  16.   Every export row has 64-char hex provenance.sig_hash
  17.   Phase 2 missing-provenance row routed to distillation_skips.jsonl
  18.   SFT quarantine populated (6 unsafe_sft_category entries)
  19.   Scratchpad/tree-split fixture row materialized
  20.   PRD drift fixture row materialized
  21.   Per-stage output_hash identical across runs (0 mismatches)
  22.   run_hash identical across runs (bit-for-bit)

CLI:
  ./scripts/distill.ts acceptance     # exits 0 on pass, 1 on fail
  ./scripts/distill.ts run-all        # full pipeline with receipts
  ./scripts/distill.ts receipts --run-id <id>

Cumulative test metrics:
  135 distillation tests pass · 0 fail · 353 expect() calls · 1411ms
  (Phase 6 adds the runtime acceptance gate, not new unit tests —
   the acceptance script IS the integration test, callable from CI.)

What this proves:
- Distillation pipeline is SAFE (contamination firewall held under
  adversarial fixture)
- Distillation pipeline is REPRODUCIBLE (identical input → bit-identical
  output across two runs)
- Distillation pipeline is GATED (every now.md invariant has a
  deterministic assertion that exits non-zero on failure)

The 6-phase distillation substrate is now training-safe. RAG (446),
SFT (351 strict-accepted), and Preference (83 paired) datasets on
real lakehouse data each carry full provenance back to source rows
through the verified Phase 2 → Phase 3 → Phase 4 chain, with Phase 5
receipts capturing every input/output sha256 + per-stage validation,
and Phase 6 proving the whole chain is gate-tight on a deterministic
fixture.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-26 23:19:56 -05:00

3.4 KiB

Raw Blame History

Phase 6 — Acceptance Gate Report

Run: 2026-04-27T04:18:55.356Z Fixture: tests/fixtures/distillation/acceptance/ Temp root: /tmp/distillation_phase6_acceptance Pipeline run_ids: acceptance-run-1-stable (first) + acceptance-run-2-stable (second / hash reproducibility)

Result: PASS ✓

Pipeline counts (first run)

collect: 14 records out · 1 skipped
score: accepted=6 rejected=4 quarantined=4
export-rag: 7 rows
export-sft: 5 rows
export-preference: 2 pairs

Invariant checks (expected vs actual)

#	Check	Expected	Actual	Status
1	receipts: all 5 stages emitted	collect,score,export-rag,export-sft,export-preference	all present	✓
2	summary.json exists	exists	exists	✓
3	summary.md exists	exists	exists	✓
4	drift.json exists	exists	exists	✓
5	every StageReceipt validates against schema	0 invalid	0 invalid	✓
6	RunSummary validates	valid	valid	✓
7	DriftReport validates	valid	valid	✓
8	SFT: ≥1 accepted record exported	>=1	5	✓
9	SFT contamination firewall: no rejected/needs_human_review	0	0	✓
10	SFT default mode: 0 partial leaks (no --include-partial used)	0	0	✓
11	RAG: 0 rejected leaks	0	0	✓
12	RAG: ≥1 partially_accepted accepted (RAG accepts partial)	>=1	2	✓
13	Preference: ≥1 valid pair exported	>=1	2	✓
14	Preference: 0 self-pairs (chosen_run_id != rejected_run_id)	0	0	✓
15	Preference: 0 identical-text pairs	0	0	✓
16	every export row has valid sha256 provenance.sig_hash	0 missing	0 missing	✓
17	Phase 2 collect: missing-provenance fixture row skipped to distillation_skips.jsonl	≥1 skip recorded	1 skip(s)	✓
18	SFT quarantine: rejected/needs_human caught at unsafe_sft_category gate	≥1	6	✓
19	scratchpad/tree-split case: fixture row materialized into evidence	found	found	✓
20	PRD drift case: fixture row materialized	found	found	✓
21	hash reproducibility: per-stage output_hash identical across runs	0 mismatches	all match	✓
22	hash reproducibility: run_hash identical	3ea12b160ee9099a...	3ea12b160ee9099a...	✓

Hash reproducibility detail

run 1 run_hash: 3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2

run 2 run_hash: 3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2

Bit-for-bit identical. Two runs of the entire pipeline on the same fixture with the same recorded_at produce the same outputs. Distillation is deterministic.

Leak prevention confirmation

SFT rows with rejected/needs_human_review quality_score: 0 (must be 0)
SFT rows with partially_accepted quality_score (default mode): 0 (must be 0; would only appear with --include-partial)
RAG rows with rejected success_score: 0 (must be 0)
Preference self-pairs (chosen_run_id == rejected_run_id): 0 (must be 0)
Preference identical-text pairs: 0 (must be 0)

What this proves

The distillation pipeline is safe, reproducible, and gated. Accepted data flows through; rejected/needs_human_review data is quarantined with reasons; preference pairs are real, not fabricated; every output traces to source via canonical sha256; running the whole pipeline twice on the same fixture produces byte-identical outputs.

3.4 KiB Raw Blame History