3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
root
|
2cf359a646 |
distillation: Phase 5 — receipts harness (system-level observability)
Forensic-grade per-stage receipts wrapping all 5 implemented pipeline
stages. Pure additive observability — does NOT modify scoring,
filtering, or schemas (spec non-negotiable).
Files (6 new):
auditor/schemas/distillation/stage_receipt.ts StageReceipt v1
auditor/schemas/distillation/run_summary.ts RunSummary v1
auditor/schemas/distillation/drift_report.ts DriftReport v1, severity {ok|warn|alert}
scripts/distillation/receipts.ts runAllWithReceipts + buildDrift + CLI
tests/distillation/receipts.test.ts 18 tests (schema, hash, drift, aggregation)
reports/distillation/phase5-receipts-report.md acceptance report
Stages wrapped:
collect (build_evidence_index → data/evidence/)
score (score_runs → data/scored-runs/)
export-rag (exports/rag/playbooks.jsonl)
export-sft (exports/sft/instruction_response.jsonl)
export-preference (exports/preference/chosen_rejected.jsonl)
Reserved (not yet implemented): extract-playbooks, index.
Output tree (per run_id):
reports/distillation/<run_id>/
collect.json score.json export-rag.json export-sft.json export-preference.json
summary.json summary.md drift.json
Test metrics: 135 distillation tests pass · 0 fail · 353 expects · 1.5s
(Phase 5 added 18; total 117→135)
Real-data run-all (run_id=78072357-835d-...):
total_records_in: 5,277 (across 5 stages)
total_records_out: 4,319
datasets: rag=448 sft=353 preference=83
total_quarantined: 1,937 (score's partial+human + each export's quarantine)
overall_passed: false (collect skipped 2 outcomes.jsonl rows missing created_at —
carry-over from Phase 2; faithfully propagated)
run_hash: 7a14d8cdd6980048a075efe97043683a4f9aabb38ec1faa8982c9887593090e0
Drift detection (second run):
prior_run_id detected automatically
severity=ok (no count or category swung >20%)
flags: ["run_hash differs from prior run"] — expected, since recorded_at
is baked into provenance and changes per run. No false alert.
Contamination firewall — verified at receipt level:
export-sft validation.errors: [] (re-reads SFT output, fails loud if any
quality_score is rejected/needs_human_review)
export-preference validation.errors: [] (re-reads, fails loud if any
chosen_run_id == rejected_run_id or chosen text == rejected text)
Invariants enforced (proven by tests + real run):
- Every stage emits ONE receipt per run (5/5 on disk)
- All receipts share run_id (uuid generated per run-all)
- aggregateIoHash is order-independent + collision-free across path/content
- Schema validators gate every receipt before write (defense in depth)
- Drift detection: pct_change > 20% → warn; new error class → warn
- Failure propagation: any stage validation.passed=false → overall_passed=false
- Self-validation: harness throws if RunSummary/DriftReport fail their own schema
CLI:
bun run scripts/distillation/receipts.ts run-all
bun run scripts/distillation/receipts.ts read --run-id <id>
Spec acceptance gate (now.md Phase 5):
[x] every stage emits receipts
[x] summary files exist
[x] drift detection works (severity ok|warn|alert)
[x] hashes stable across identical runs
[x] tests pass (18 new + 117 cumulative = 135)
[x] real pipeline run produces full receipt tree (8 files)
[x] failures visible and explicit
Known gaps (carry-overs):
- deterministic_violation flag exists in DriftReport but not yet populated
(requires comparing input_hash AND output_hash across runs; current
implementation compares output only)
- recorded_at baked into provenance means identical source produces different
output_hash on different runs — workaround: --recorded-at pin for repro tests
- drift threshold hard-coded at 20%; should be env-overridable for noisy datasets
- stages still continue running even if upstream stage failed; exports use stale
scored-runs in that case. Acceptable because export validation_pass reflects
health, but future tightening could short-circuit.
Phase 6 (acceptance gate suite) unblocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
68b6697bcb |
distillation: Phase 4 — dataset export layer
Some checks failed
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Build the contamination firewall: RAG, SFT, and Preference exporters
that turn scored evidence into clean training datasets without
leaking rejected, unvalidated, hallucinated, or provenance-free
records.
Files (8 new + 4 schema updates):
scripts/distillation/quarantine.ts shared QuarantineWriter, 11-reason taxonomy
scripts/distillation/export_rag.ts RAG exporter (--include-review opt-in)
scripts/distillation/export_sft.ts SFT exporter (--include-partial opt-in, SFT_NEVER constant)
scripts/distillation/export_preference.ts preference exporter, same task_id pairing
scripts/distillation/distill.ts CLI dispatcher (build-evidence/score/export-*)
tests/distillation/exports.test.ts 15 contamination-firewall tests
reports/distillation/phase4-export-report.md acceptance report
Schema field-name alignment with now.md:
rag_sample.ts +source_category, exported_at→created_at
sft_sample.ts +id, exported_at→created_at, partially_accepted at schema (CLI gates)
preference_sample.ts +id, source_run_ids→chosen_run_id+rejected_run_id, +created_at
Test metrics: 117 distillation tests pass · 0 fail · 315 expects · 327ms
Real-data export run (1052 scored input rows):
RAG: 446 exported (351 acc + 95 partial), 606 quarantined
SFT: 351 exported (all 'accepted'), 701 quarantined
Preference: 83 pairs exported, 16 quarantined
CONTAMINATION FIREWALL — verified held on real data:
- SFT output: 351/351 quality_score='accepted' (ZERO leaked)
- RAG output: 351 acc + 95 partial (ZERO rejected leaked)
- Preference: 0 self-pairs (chosen_run_id != rejected_run_id)
- 536 rejected+needs_human_review records caught at unsafe_sft_category
gate, exact match to scored-runs forbidden-category total
Defense in depth (the firewall is two layers, not one):
1. Schema layer (Phase 1): SftSample.quality_score enum forbids
rejected/needs_human at write time
2. Exporter layer: SFT_NEVER constant in export_sft.ts checks
category before synthesis. Even if synthesis produced a row
with quality_score=rejected, validateSftSample would reject it.
Quarantine reasons (11): missing_provenance, missing_source_run_id,
empty_content, schema_violation, unsafe_sft_category,
unsafe_rag_category, invalid_preference_pairing,
hallucinated_file_path, duplicate_id, self_pairing,
category_disallowed.
Bug surfaced + fixed during testing: module-level evidenceCache
shared state across test runs (tests wipe TMP, cache holds stale
empty Map). Moved cache to per-call scope. Same pattern bit Phase 2
materializer would have hit if its tests had multiple runs sharing
state — preventive fix.
Pairing logic v1: same task_id with category gap. accepted×rejected
preferred, accepted×partially_accepted as fallback. MAX_PAIRS_PER_TASK=5
cap prevents one hot task from dominating. Future: cross-source
pairing (scrum_reviews chosen vs observer_reviews rejected on same
file) to grow dataset beyond 83.
CLI: ./scripts/distill.ts {build-evidence|score|export-rag|export-sft|export-preference|export-all|health}
Flags: --dry-run, --include-partial (SFT only), --include-review (RAG only)
Carry-overs to Phase 5 (Receipts Harness):
- Each exporter currently writes results but no per-stage receipt.json.
Phase 5 wraps build_evidence_index + score_runs + export_* in a
withReceipt() helper that captures git_sha + sha256 of inputs/outputs
+ record_counts + validation_pass.
- reports/distillation/latest.md aggregating most-recent run of each stage.
Carry-overs to Phase 3 v2:
- mode_experiments scoring (168 needs_human_review): derive markers from
validation_results.grounded_fraction
- extraction-class JOIN: distilled_*/audit_facts/observer_escalations
→ JOIN to verdict-bearing parent by task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
27b1d27605 |
distillation: Phase 0 recon + Phase 1 schemas + Phase 2 transforms scaffold
Some checks failed
lakehouse/auditor 9 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Phase 0 — docs/recon/local-distillation-recon.md
Inventories the 23 KB JSONL streams + 20 vector corpora + auditor's
kb_index.ts as substrate for the now.md distillation pipeline. Maps
spec modules to existing producers, identifies real gaps, lists 9
schemas to formalize. ZERO implementation in recon — gating doc only.
Phase 1 — auditor/schemas/distillation/
9 schemas + foundation types + 48 tests passing in 502ms:
types.ts shared validators + canonicalSha256
evidence_record.ts EVIDENCE_SCHEMA_VERSION=1, ModelRole enum
scored_run.ts 4 categories pinned, anchor_grounding ∈ [0,1]
receipt.ts git_sha 40-char, sha256 file refs, validation_pass:bool
playbook.ts non-empty source_run_ids + acceptance_criteria
scratchpad_summary.ts validation_status enum, hash sha256
model_ledger.ts success_rate ∈ [0,1], sample_count ≥ 1
rag_sample.ts success_score ∈ {accepted, partially_accepted}
sft_sample.ts quality_score MUST be 'accepted' (no leak)
preference_sample.ts chosen != rejected, source_run_ids must differ
evidence_record.test.ts 10 tests, JSON-fixture round-trip
schemas.test.ts 30 tests, inline fixtures
realdata.test.ts 8 tests, real-JSONL probe
Real-data validation probe (one of the 3 notables from recon):
46 rows across 7 sources, 100% pass. distilled_facts/procedures alive.
Report at data/_kb/realdata_validation_report.md (also written by the
test). Confirms schema fits existing producers without migration.
Phase 2 scaffold — scripts/distillation/transforms.ts
Promoted PROBES from realdata.test.ts into a real TRANSFORMS array
covering 12 source streams (8 Tier 1 validated + 4 Tier 2 from
recon's untested-streams list). Pure functions: no I/O, no model
calls, no clock reads. Caller supplies recorded_at + sig_hash so
materializer is deterministic by construction.
Spec non-negotiables enforced at schema layer (defense in depth):
- provenance{source_file, sig_hash, recorded_at} required everywhere
- schema_version mismatch hard-rejects (forward-compat gate)
- SFT no-leak: validateSftSample REJECTS partially_accepted, rejected,
needs_human_review — three explicit tests
- Every score has WHY (reasons non-empty)
- Every playbook traces to source (source_run_ids non-empty)
- Every preference has WHY (reason non-empty)
- Receipts substantive (git_sha 40-char, sha256 64-char, validation_pass:bool)
Branch carries uncommitted auditor rebuild work (mode.rs + modes.toml
+ inference.ts + static.ts) blocked on upstream Ollama Cloud kimi-k2
500 ISE; held pending recon-driven design decisions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|