3d068681f5
Some checks failed
lakehouse/auditor 17 blocking issues: cloud: claim not backed — "Verified end-to-end via playwright on devop.live/lakehouse:"
Phase 6 acceptance + Phase 8 full-audit reports re-run; bit-for-bit reproducibility property still holds (run 1 hash == run 2 hash), just at a new value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3.6 KiB
3.6 KiB
Phase 8 — Full System Audit Report
Run: 2026-04-27T15:43:38.021Z
Git commit: ca7375ea2b
Baseline: 2026-04-27T10:31:44.043Z (d11632a6fa)
Result: PASS ✓
Per-phase summary
| Phase | Checks | Required | Required-Pass | Notes |
|---|---|---|---|---|
| 0 | 2 | 1 | 1/1 | ✓ pass |
| 1 | 1 | 1 | 1/1 | ✓ pass |
| 2 | 2 | 1 | 1/1 | ✓ pass |
| 3 | 2 | 1 | 1/1 | ✓ pass |
| 4 | 5 | 5 | 5/5 | ✓ pass |
| 5 | 5 | 4 | 4/4 | ✓ pass |
| 6 | 1 | 1 | 1/1 | ✓ pass |
| 7 | 4 | 2 | 2/2 | ✓ pass |
Detailed checks
| # | Phase | Check | Required | Expected | Actual | Status |
|---|---|---|---|---|---|---|
| 1 | P0 | recon doc exists | Y | docs/recon/local-distillation-recon.md present | present | ✓ |
| 2 | P0 | tier-1 source streams present | — | all 4 tier-1 jsonls on disk | all present | ✓ |
| 3 | P1 | schema validators pass on fixtures | Y | ≥40 tests, 0 fail | 51 pass, 0 fail | ✓ |
| 4 | P2 | materializer dry-run completes | Y | >=1 row from each tier-1 source | 1139 read · 82 written · 2 skipped | ✓ |
| 5 | P2 | tier-1 sources each materialize ≥1 row | — | 4/4: distilled_facts, scrum_reviews, audit_facts, mode_experiments | 1/4 hit (mode_experiments) | ✓ |
| 6 | P3 | on-disk scored-runs distribution non-empty | Y | >=1 accepted | acc=386 part=132 rej=57 hum=480 | ✓ |
| 7 | P3 | scored-runs distribution sums positive | — | >0 total | 1055 total | ✓ |
| 8 | P4 | SFT contamination firewall: 0 forbidden quality_scores | Y | 0 | 0 | ✓ |
| 9 | P4 | RAG firewall: 0 rejected leaks | Y | 0 | 0 | ✓ |
| 10 | P4 | Preference: 0 self-pairs (chosen_run_id != rejected_run_id) | Y | 0 | 0 | ✓ |
| 11 | P4 | Preference: 0 identical-text pairs | Y | 0 | 0 | ✓ |
| 12 | P4 | every export row carries valid sha256 provenance.sig_hash | Y | 0 missing | 0 missing | ✓ |
| 13 | P5 | latest run (3fa51d66-784c-4c7d-843d-6c48328a608c) has all 5 stage receipts | Y | collect,score,export-rag,export-sft,export-preference | all present | ✓ |
| 14 | P5 | every stage receipt validates against schema | Y | 0 invalid | 0 invalid | ✓ |
| 15 | P5 | RunSummary validates | Y | valid | valid | ✓ |
| 16 | P5 | summary.git_commit is 40-char hex | — | match | 68b6697bcb38... (HEAD: ca7375ea2b17...) | ✓ |
| 17 | P5 | run_hash is sha256 | Y | /^[0-9a-f]{64}$/ | 2336b96c3638982d... | ✓ |
| 18 | P6 | acceptance gate passes 22/22 invariants on fixture | Y | PASS — 22/22 | 22/22 (exit=0) | ✓ |
| 19 | P7 | replay validation passes on 3/3 dry-run sample tasks | Y | 3/3 | 3/3 | ✓ |
| 20 | P7 | replay retrieval surfaces ≥1 playbook on each task (when corpus present) | — | ≥1 task with retrieval | 3/3 | ✓ |
| 21 | P7 | escalation loop guard: no path > 2 models | Y | 0 loops | 0 | ✓ |
| 22 | P7 | replay_runs.jsonl populated by audit run | — | exists with ≥3 rows added | 27 rows total | ✓ |
Drift vs prior baseline
| Metric | Baseline | Current | Δ% | Flag |
|---|---|---|---|---|
| p2_evidence_rows | 25 | 82 | 228% | warn |
| p2_evidence_skips | 2 | 2 | 0% | ok |
| p3_accepted | 386 | 386 | 0% | ok |
| p3_partial | 132 | 132 | 0% | ok |
| p3_rejected | 57 | 57 | 0% | ok |
| p3_human | 480 | 480 | 0% | ok |
| p4_rag_rows | 448 | 448 | 0% | ok |
| p4_sft_rows | 353 | 353 | 0% | ok |
| p4_pref_pairs | 83 | 83 | 0% | ok |
| p4_total_quarantined | 1325 | 1325 | 0% | ok |
1 metric(s) drifted >20% from baseline. Investigate before treating outputs as stable.
System health status
All required Phase 0-7 invariants hold. The distillation system is correct, stable, and reproducible at this commit.