1b433a9308
End-to-end fixture-driven gate. Runs the entire pipeline (collect →
score → export-rag → export-sft → export-preference) on a deterministic
fixture, asserts 22 invariants, runs a SECOND time with the same
recorded_at, and verifies hash reproducibility. Exits non-zero on any
failure. Pure observability — no scoring/filtering/schema changes.
Files (3 new + 1 modified + 6 fixture jsonls):
scripts/distillation/acceptance.ts 330 lines, runner + 22 checks
reports/distillation/phase6-acceptance-report.md autogenerated by run
scripts/distillation/distill.ts +run-all, +receipts, +acceptance subcommands
tests/fixtures/distillation/acceptance/data/_kb/
scrum_reviews.jsonl 5 rows (accepted/partial/needs_human/scratchpad/missing-provenance)
audits.jsonl 3 rows (info/high+PRD-drift/medium severity)
auto_apply.jsonl 2 rows (committed, build_red_reverted)
contract_analyses.jsonl 2 rows (accept, reject)
observer_reviews.jsonl 2 rows (accept, reject — pair candidates)
distilled_facts.jsonl 1 extraction-class row
Spec cases covered (now.md Phase 6):
✓ accepted — Row #1 scrum, #6 audit-info, #11 contract-accept, #14 obs-accept
✓ partially_accepted — Row #2 scrum (3 attempts), #8 audit-medium
✓ rejected — #7 audit-high, #10 auto_apply build_red, #12 contract-reject, #15 obs-reject
✓ needs_human_review — #3 scrum (no markers), #13 distilled extraction-class
✓ missing provenance — Row #5 scrum (no reviewed_at) → routed to skips
✓ valid preference pair — observer_reviews accept+reject on same file
✓ invalid preference pair — quarantine reasons populated when generated
✓ scratchpad / tree-split — Row #4 scrum tree_split_fired=true with multi-shard text
✓ PRD drift — Row #7 audit severity=high, topic="PRD drift: circuit breaker shipped claim"
Acceptance run results (run_id: acceptance-run-1-stable):
22/22 invariants PASS
Pipeline counts:
collect: 14 records out, 1 skipped (missing-provenance fixture)
score: accepted=6 rejected=4 quarantined=4
export-rag: 7 rows (5 acc + 2 partial, ZERO rejected)
export-sft: 5 rows (all 'accepted', ZERO partial without --include-partial)
export-preference: 2 pairs (zero self-pairs, zero identical-text)
Hash reproducibility — bit-for-bit identical:
run_hash: 3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2
Two runs of the entire pipeline on the same fixture with the same
recorded_at produce byte-identical outputs.
The 22 invariants:
1-4. Receipts + summary.json + summary.md + drift.json exist
5-7. StageReceipt + RunSummary + DriftReport schemas all valid
8-10. SFT contains accepted only — no rejected/needs_human/partial leak
11-12. RAG contains accepted+partial — zero rejected
13-15. Preference: ≥1 pair, zero self-pairs, zero identical text
16. Every export row has 64-char hex provenance.sig_hash
17. Phase 2 missing-provenance row routed to distillation_skips.jsonl
18. SFT quarantine populated (6 unsafe_sft_category entries)
19. Scratchpad/tree-split fixture row materialized
20. PRD drift fixture row materialized
21. Per-stage output_hash identical across runs (0 mismatches)
22. run_hash identical across runs (bit-for-bit)
CLI:
./scripts/distill.ts acceptance # exits 0 on pass, 1 on fail
./scripts/distill.ts run-all # full pipeline with receipts
./scripts/distill.ts receipts --run-id <id>
Cumulative test metrics:
135 distillation tests pass · 0 fail · 353 expect() calls · 1411ms
(Phase 6 adds the runtime acceptance gate, not new unit tests —
the acceptance script IS the integration test, callable from CI.)
What this proves:
- Distillation pipeline is SAFE (contamination firewall held under
adversarial fixture)
- Distillation pipeline is REPRODUCIBLE (identical input → bit-identical
output across two runs)
- Distillation pipeline is GATED (every now.md invariant has a
deterministic assertion that exits non-zero on failure)
The 6-phase distillation substrate is now training-safe. RAG (446),
SFT (351 strict-accepted), and Preference (83 paired) datasets on
real lakehouse data each carry full provenance back to source rows
through the verified Phase 2 → Phase 3 → Phase 4 chain, with Phase 5
receipts capturing every input/output sha256 + per-stage validation,
and Phase 6 proving the whole chain is gate-tight on a deterministic
fixture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
64 lines
3.4 KiB
Markdown
64 lines
3.4 KiB
Markdown
# Phase 6 — Acceptance Gate Report
|
|
|
|
**Run:** 2026-04-27T04:18:55.356Z
|
|
**Fixture:** `tests/fixtures/distillation/acceptance/`
|
|
**Temp root:** `/tmp/distillation_phase6_acceptance`
|
|
**Pipeline run_ids:** `acceptance-run-1-stable` (first) + `acceptance-run-2-stable` (second / hash reproducibility)
|
|
|
|
## Result: **PASS** ✓
|
|
|
|
## Pipeline counts (first run)
|
|
|
|
- collect: 14 records out · 1 skipped
|
|
- score: accepted=6 rejected=4 quarantined=4
|
|
- export-rag: 7 rows
|
|
- export-sft: 5 rows
|
|
- export-preference: 2 pairs
|
|
|
|
## Invariant checks (expected vs actual)
|
|
|
|
| # | Check | Expected | Actual | Status |
|
|
|---|---|---|---|---|
|
|
| 1 | receipts: all 5 stages emitted | collect,score,export-rag,export-sft,export-preference | all present | ✓ |
|
|
| 2 | summary.json exists | exists | exists | ✓ |
|
|
| 3 | summary.md exists | exists | exists | ✓ |
|
|
| 4 | drift.json exists | exists | exists | ✓ |
|
|
| 5 | every StageReceipt validates against schema | 0 invalid | 0 invalid | ✓ |
|
|
| 6 | RunSummary validates | valid | valid | ✓ |
|
|
| 7 | DriftReport validates | valid | valid | ✓ |
|
|
| 8 | SFT: ≥1 accepted record exported | >=1 | 5 | ✓ |
|
|
| 9 | SFT contamination firewall: no rejected/needs_human_review | 0 | 0 | ✓ |
|
|
| 10 | SFT default mode: 0 partial leaks (no --include-partial used) | 0 | 0 | ✓ |
|
|
| 11 | RAG: 0 rejected leaks | 0 | 0 | ✓ |
|
|
| 12 | RAG: ≥1 partially_accepted accepted (RAG accepts partial) | >=1 | 2 | ✓ |
|
|
| 13 | Preference: ≥1 valid pair exported | >=1 | 2 | ✓ |
|
|
| 14 | Preference: 0 self-pairs (chosen_run_id != rejected_run_id) | 0 | 0 | ✓ |
|
|
| 15 | Preference: 0 identical-text pairs | 0 | 0 | ✓ |
|
|
| 16 | every export row has valid sha256 provenance.sig_hash | 0 missing | 0 missing | ✓ |
|
|
| 17 | Phase 2 collect: missing-provenance fixture row skipped to distillation_skips.jsonl | ≥1 skip recorded | 1 skip(s) | ✓ |
|
|
| 18 | SFT quarantine: rejected/needs_human caught at unsafe_sft_category gate | ≥1 | 6 | ✓ |
|
|
| 19 | scratchpad/tree-split case: fixture row materialized into evidence | found | found | ✓ |
|
|
| 20 | PRD drift case: fixture row materialized | found | found | ✓ |
|
|
| 21 | hash reproducibility: per-stage output_hash identical across runs | 0 mismatches | all match | ✓ |
|
|
| 22 | hash reproducibility: run_hash identical | 3ea12b160ee9099a... | 3ea12b160ee9099a... | ✓ |
|
|
|
|
## Hash reproducibility detail
|
|
|
|
run 1 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2`
|
|
|
|
run 2 run_hash: `3ea12b160ee9099a3c52fe6e7fffd3076de7920d2704d24c789260d63cb1a5a2`
|
|
|
|
**Bit-for-bit identical.** Two runs of the entire pipeline on the same fixture with the same `recorded_at` produce the same outputs. Distillation is deterministic.
|
|
|
|
## Leak prevention confirmation
|
|
|
|
- SFT rows with rejected/needs_human_review quality_score: **0** (must be 0)
|
|
- SFT rows with partially_accepted quality_score (default mode): **0** (must be 0; would only appear with --include-partial)
|
|
- RAG rows with rejected success_score: **0** (must be 0)
|
|
- Preference self-pairs (chosen_run_id == rejected_run_id): **0** (must be 0)
|
|
- Preference identical-text pairs: **0** (must be 0)
|
|
|
|
## What this proves
|
|
|
|
The distillation pipeline is **safe, reproducible, and gated**. Accepted data flows through; rejected/needs_human_review data is quarantined with reasons; preference pairs are real, not fabricated; every output traces to source via canonical sha256; running the whole pipeline twice on the same fixture produces byte-identical outputs.
|