68b6697bcb
Some checks failed
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Build the contamination firewall: RAG, SFT, and Preference exporters
that turn scored evidence into clean training datasets without
leaking rejected, unvalidated, hallucinated, or provenance-free
records.
Files (8 new + 4 schema updates):
scripts/distillation/quarantine.ts shared QuarantineWriter, 11-reason taxonomy
scripts/distillation/export_rag.ts RAG exporter (--include-review opt-in)
scripts/distillation/export_sft.ts SFT exporter (--include-partial opt-in, SFT_NEVER constant)
scripts/distillation/export_preference.ts preference exporter, same task_id pairing
scripts/distillation/distill.ts CLI dispatcher (build-evidence/score/export-*)
tests/distillation/exports.test.ts 15 contamination-firewall tests
reports/distillation/phase4-export-report.md acceptance report
Schema field-name alignment with now.md:
rag_sample.ts +source_category, exported_at→created_at
sft_sample.ts +id, exported_at→created_at, partially_accepted at schema (CLI gates)
preference_sample.ts +id, source_run_ids→chosen_run_id+rejected_run_id, +created_at
Test metrics: 117 distillation tests pass · 0 fail · 315 expects · 327ms
Real-data export run (1052 scored input rows):
RAG: 446 exported (351 acc + 95 partial), 606 quarantined
SFT: 351 exported (all 'accepted'), 701 quarantined
Preference: 83 pairs exported, 16 quarantined
CONTAMINATION FIREWALL — verified held on real data:
- SFT output: 351/351 quality_score='accepted' (ZERO leaked)
- RAG output: 351 acc + 95 partial (ZERO rejected leaked)
- Preference: 0 self-pairs (chosen_run_id != rejected_run_id)
- 536 rejected+needs_human_review records caught at unsafe_sft_category
gate, exact match to scored-runs forbidden-category total
Defense in depth (the firewall is two layers, not one):
1. Schema layer (Phase 1): SftSample.quality_score enum forbids
rejected/needs_human at write time
2. Exporter layer: SFT_NEVER constant in export_sft.ts checks
category before synthesis. Even if synthesis produced a row
with quality_score=rejected, validateSftSample would reject it.
Quarantine reasons (11): missing_provenance, missing_source_run_id,
empty_content, schema_violation, unsafe_sft_category,
unsafe_rag_category, invalid_preference_pairing,
hallucinated_file_path, duplicate_id, self_pairing,
category_disallowed.
Bug surfaced + fixed during testing: module-level evidenceCache
shared state across test runs (tests wipe TMP, cache holds stale
empty Map). Moved cache to per-call scope. Same pattern bit Phase 2
materializer would have hit if its tests had multiple runs sharing
state — preventive fix.
Pairing logic v1: same task_id with category gap. accepted×rejected
preferred, accepted×partially_accepted as fallback. MAX_PAIRS_PER_TASK=5
cap prevents one hot task from dominating. Future: cross-source
pairing (scrum_reviews chosen vs observer_reviews rejected on same
file) to grow dataset beyond 83.
CLI: ./scripts/distill.ts {build-evidence|score|export-rag|export-sft|export-preference|export-all|health}
Flags: --dry-run, --include-partial (SFT only), --include-review (RAG only)
Carry-overs to Phase 5 (Receipts Harness):
- Each exporter currently writes results but no per-stage receipt.json.
Phase 5 wraps build_evidence_index + score_runs + export_* in a
withReceipt() helper that captures git_sha + sha256 of inputs/outputs
+ record_counts + validation_pass.
- reports/distillation/latest.md aggregating most-recent run of each stage.
Carry-overs to Phase 3 v2:
- mode_experiments scoring (168 needs_human_review): derive markers from
validation_results.grounded_fraction
- extraction-class JOIN: distilled_*/audit_facts/observer_escalations
→ JOIN to verdict-bearing parent by task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61 lines
2.5 KiB
TypeScript
61 lines
2.5 KiB
TypeScript
// PreferenceSample — entry in exports/preference/chosen_rejected.jsonl.
|
|
// Source: real disagreements (audit_discrepancies, scrum ladder retries).
|
|
// Validator pins: chosen != rejected, both source_run_ids present, reason
|
|
// is non-empty. No synthesized preferences.
|
|
import {
|
|
ValidationResult, requireString, requireIsoTimestamp, requireProvenance,
|
|
} from "./types";
|
|
|
|
export const PREFERENCE_SAMPLE_SCHEMA_VERSION = 1;
|
|
|
|
export interface PreferenceSample {
|
|
schema_version: number;
|
|
id: string;
|
|
prompt: string;
|
|
chosen: string;
|
|
rejected: string;
|
|
reason: string; // why chosen > rejected — must be non-empty
|
|
chosen_run_id: string;
|
|
rejected_run_id: string;
|
|
created_at: string;
|
|
provenance: { source_file: string; line_offset?: number; sig_hash: string; recorded_at: string };
|
|
}
|
|
|
|
export function validatePreferenceSample(input: unknown): ValidationResult<PreferenceSample> {
|
|
const errors: string[] = [];
|
|
if (typeof input !== "object" || input === null) return { valid: false, errors: ["expected object"] };
|
|
const r = input as Record<string, unknown>;
|
|
let ok = true;
|
|
|
|
if (r.schema_version !== PREFERENCE_SAMPLE_SCHEMA_VERSION) {
|
|
errors.push(`schema_version: expected ${PREFERENCE_SAMPLE_SCHEMA_VERSION}, got ${JSON.stringify(r.schema_version)}`);
|
|
ok = false;
|
|
}
|
|
ok = requireString(r.id, "id", errors) && ok;
|
|
ok = requireString(r.prompt, "prompt", errors) && ok;
|
|
ok = requireString(r.chosen, "chosen", errors) && ok;
|
|
ok = requireString(r.rejected, "rejected", errors) && ok;
|
|
ok = requireString(r.reason, "reason", errors) && ok;
|
|
ok = requireString(r.chosen_run_id, "chosen_run_id", errors) && ok;
|
|
ok = requireString(r.rejected_run_id, "rejected_run_id", errors) && ok;
|
|
ok = requireIsoTimestamp(r.created_at, "created_at", errors) && ok;
|
|
ok = requireProvenance(r.provenance, "provenance", errors) && ok;
|
|
|
|
// Self-pairing guard.
|
|
if (r.chosen === r.rejected && typeof r.chosen === "string") {
|
|
errors.push("chosen and rejected must differ — preference data needs a real disagreement");
|
|
ok = false;
|
|
}
|
|
if (r.chosen_run_id === r.rejected_run_id && typeof r.chosen_run_id === "string") {
|
|
errors.push("chosen_run_id and rejected_run_id must differ — same run can't disagree with itself");
|
|
ok = false;
|
|
}
|
|
if (typeof r.reason === "string" && (r.reason as string).trim().length === 0) {
|
|
errors.push("reason: must be non-whitespace (every preference needs WHY chosen > rejected)");
|
|
ok = false;
|
|
}
|
|
|
|
if (!ok) return { valid: false, errors };
|
|
return { valid: true, value: r as unknown as PreferenceSample };
|
|
}
|