2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
root
|
d77622fc6b |
distillation: fix 7 grounding bugs found by Kimi audit
Kimi For Coding (api.kimi.com, kimi-for-coding) ran a forensic audit on
distillation v1.0.0 with full file content. 7/7 flags verified real on
grep. Substrate now matches what v1.0.0 claimed: deterministic, no
schema bypasses, Rust tests compile.
Fixes:
- mode.rs:1035,1042 matrix_corpus Some/None -> vec![..]/vec![]; cargo
check --tests now compiles (was silently broken;
only bun tests were running)
- scorer.ts:30 SCORER_VERSION env override removed - identical
input now produces identical version stamp, not
env-dependent drift
- transforms.ts:181 auto_apply wall-clock fallback (new Date()) ->
deterministic recorded_at fallback
- replay.ts:378 recorded_run_id Date.now() -> sha256(recorded_at);
replay rows now reproducible given recorded_at
- receipts.ts:454,495 input_hash_match hardcoded true was misleading
telemetry; bumped DRIFT_REPORT_SCHEMA_VERSION 1->2,
field is now boolean|null with honest null when
not computed at this layer
- score_runs.ts:89-100,159 dedup keyed only on sig_hash made
scorer-version bumps invisible. Composite
sig_hash:scorer_version forces re-scoring
- export_sft.ts:126 (ev as any).contractor bypass emitted "<contractor>"
placeholder for every contract_analyses SFT row.
Added typed EvidenceRecord.metadata bucket;
transforms.ts populates metadata.contractor;
exporter reads typed value
Verification (all green):
cargo check -p gateway --tests compiles
bun test tests/distillation/ 145 pass / 0 fail
bun acceptance 22/22 invariants
bun audit-full 16/16 required checks
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
681f39d5fa |
distillation: Phase 7 — replay-driven local model bootstrapping
Some checks failed
lakehouse/auditor 13 blocking issues: cloud: claim not backed — "probes; multi-hour outage). deepseek is the proven drop-in from"
Runtime layer that takes a task → retrieves matching playbooks/RAG
records → builds a structured context bundle → feeds it to a LOCAL
model (qwen3.5:latest, ~7B class) → validates output → escalates only
when needed → logs the full run as new evidence. NOT model training.
Pure runtime behavior shaping via retrieval against the Phase 0-6
distillation substrate.
Files (3 new + 1 modified):
scripts/distillation/replay.ts ~370 lines
tests/distillation/replay.test.ts 10 tests, 19 expects
scripts/distillation/distill.ts +replay subcommand
reports/distillation/phase7-replay-report.md
Test metrics: 145 cumulative distillation tests pass · 0 fail · 372 expects · 618ms
Real-data A/B on 3 tasks (same qwen3.5:latest local model, with vs
without retrieval) — proves the spec claim "local model improves
with retrieval":
Task 1 "Audit phase 38 provider routing":
WITH retrieval: cited V1State, openrouter, /v1/chat, ProviderAdapter,
PRD.md line ranges — REAL Lakehouse internals
WITHOUT retrieval: invented "P99999, Z99999 placeholder codes" and
"production routing table" — pure fabrication
Task 2 "Verify pr_audit mode wired":
WITH: correct crates/gateway/src/main.rs path + lakehouse_answers_v1
WITHOUT: same assertion, no proof, asserts confidently
Task 3 "Audit phase 40 PRD circuit breaker drift":
WITH: anchored on the actual audit finding "no breaker class found"
WITHOUT: invented "0.0% failure rate vs 5.0% threshold" and signed
off as PASS on broken code — exact failure mode the
distillation pipeline was built to prevent
Both runs passed the structural validation gate (length, no hedges,
checklist token overlap) — the difference is grounding, supplied by
the retrieval layer pulling from exports/rag/playbooks.jsonl (446
records from earlier Phase 4 export).
Architecture:
jaccard token overlap against rag corpus → top-K (default 8) split
into accepted exemplars (top 3) + partial-warnings (top 2) + extracted
validation_steps (lines starting verify|check|assert|ensure|confirm)
→ prompt assembly → qwen3.5:latest via /v1/chat (or OpenRouter
for namespaced/free models) → deterministic validation gate →
escalation to deepseek-v3.1:671b on fail with --allow-escalation
→ log to data/_kb/replay_runs.jsonl
Spec invariants enforced:
- never bypass retrieval (--no-retrieval is explicit baseline, not default)
- never discard provenance (task_hash + rag_ids + full bundle logged)
- never allow free-form hallucinated output (validation gate is
deterministic code, never an LLM)
- log every run as new evidence (replay_run.v1 schema, append-only
to data/_kb/replay_runs.jsonl)
CLI:
./scripts/distill replay --task "<input>" [--local-only]
[--allow-escalation]
[--no-retrieval]
What this unlocks:
The substrate for "small-model bootstrapping" and "local inference
dominance" J flagged after Phase 5. Phase 8+ closes the loop:
schedule replay runs on common tasks, score outputs, feed accepted
ones back into corpus, measure escalation rate decreasing over time.
Known limitations (documented in report):
- Validation gate is structural not semantic (catches hedges/empty
but not plausible-wrong). Phase 13 wiring: run auditor against
every replay output.
- Retrieval is jaccard keyword. Works at 446 corpus, scale via
/vectors/search HNSW retrieval once corpus crosses ~10k.
- Convergence claim is architectural (deterministic retrieval +
low-temp call); longitudinal empirical study is Phase 8+.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|