9-run empirical test showed 20 of 27 audit_lessons signatures were
singletons (count=1) — the cloud producing slightly-different summary
phrasings for the SAME underlying claim on each audit, each hashing
to a fresh signature. That's the creep J flagged — not explosive,
but steady ~2 new sigs per run, unbounded over hundreds of runs.
Root cause: temperature=0.2 + think=true was letting variable prose
leak into the classification output. Fix: temp=0 (greedy sample →
identical input yields identical output on same model version),
think=false (no reasoning trace variance), max_tokens 3000→1500
(tighter bound prevents tail wander).
The compounding policy itself was validated by the 9 runs:
- 7 recurring claims (the legitimate signals) all at conf 0.08-0.20
- ratingSeverity() correctly held them at info (below 0.3 threshold)
- cross-PR signal test separately confirmed conf=1.00 → sev=block
Also: LH_AUDIT_RUNS env so the test can validate with smaller N.
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Phase 1 — definition-layer over append-only JSONL scratchpads.
auditor/kb_index.ts is the single shared aggregator:
aggregate<T>(jsonlPath, { keyFn, scopeFn, checkFn, tailLimit })
→ Map<signature, {count, distinct_scopes, confidence,
first_seen, last_seen, representative_summary, ...}>
ratingSeverity(agg) — confidence × count severity policy shared
across all KB readers. Kills the "same unfixed PR inflates its
own recurrence score" failure mode by design: confidence =
distinct_scopes/count, so same-scope noise stays below the 0.3
escalation threshold no matter how many times it repeats.
checkAuditLessons now routes through aggregate + ratingSeverity.
Net effect: the recurrence detector's bespoke Map/Set bookkeeping is
gone; same behavior, shared discipline, reusable by scrum/observer.
Also: symbolsExistInRepo now skips files >500KB so the audit can't
get stuck slurping a fixture.
Phase 2 — nine-consecutive audit runner.
tests/real-world/nine_consecutive_audits.ts pushes 9 empty commits,
waits for each verdict, captures the audit_lessons aggregate state
after each run, reports:
- sig_count trajectory (should stabilize, not grow linearly)
- max_count trajectory (same-signature repeat rate)
- max_confidence trajectory (must stay LOW on same-PR noise)
- verdict_stable across runs (must NOT oscillate)
This is the empirical proof that the KB compounds favorably:
noise doesn't escalate itself, and signal stays distinguishable.
Unit-tested both failure modes: same-PR × 9 repeats = conf=0.11
(info); cross-PR × 5 distinct = conf=1.00 (block). The rating
function correctly discriminates.
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Observed on PR #8 audit (de11ac4): 7 warn findings, all from the
cloud inference check. Investigation showed two distinct bug classes
that weren't "ship bad code", they were "auditor misreads the diff":
1. Cloud flagged "X not defined in this diff / missing implementation"
for symbols like `tailJsonl` and `stubFinding` that ARE defined —
just not in the added lines of this diff. Fix: extract candidate
symbols from the cloud's gap summary, grep the repo for their
definitions (function/const/let/def/class/struct/enum/trait/fn).
If every named symbol resolves, drop the finding; if some do,
demote to info with the resolution in evidence.
2. Cloud flagged runtime metrics like "58 cloud calls, 306s
end-to-end" as unbacked claims. These are empirical outputs
from running the test, not things a static diff can prove.
Fix: claim_parser now has an `empirical` strength class
matching iteration counts, cloud-call counts, duration metrics,
attempt counts, tier-count phrases. Inference drops empirical
claims from its cloud prompt (verifiable[] subset only) and
claim-index mapping uses verifiable[] so cloud responses still
line up.
Added `claims_empirical` to audit metrics so the verdict is
introspectable: how many claims WERE runtime-only vs how many
are diff-verifiable?
Verified: unit tests confirm empirical classification on 5
sample commit messages; symbol resolver found both false-positive
symbols (tailJsonl + stubFinding) and correctly skipped a known-
fake symbol.
Adds State section entries for the two KB files that close the
feedback loop: audit_lessons.jsonl (findings → recurrence detector)
and scrum_reviews.jsonl (scrum output → kb_query surfacing).
Touch-commit to trigger re-audit on fresh SHA with the restarted
auditor (which now has the fix-loaded code).
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Two changes that fell out of running the auto-loop for real on PR #8:
1. The systemd auditor blocked PR #8 on 'unimplemented!()' / 'todo!()'
in tests/real-world/hard_task_escalation.ts — but those strings are
the rubric itself, not macro calls. Added isInsideQuotedString()
detection in static.ts: BLOCK_PATTERNS now skip matches that fall
inside double-quoted / single-quoted / backtick string literals on
the added line. WARN/INFO patterns still run — a TODO comment in
a string is still a valid signal.
2. Verdicts were being persisted to disk but never fed back as
learning signal. Added appendAuditLessons() — every block/warn
finding writes a JSONL row to data/_kb/audit_lessons.jsonl with a
path-agnostic signature (strips file paths, line numbers, commit
hashes) so the SAME class of finding on DIFFERENT files dedups to
one signature.
kb_query now tails audit_lessons.jsonl and emits recurrence
findings: 2 distinct PRs hit a signature = info, 3-4 = warn, 5+ =
block. Severity ramps on distinct-PR count, not total rows, so a
single unfixed PR being re-audited doesn't inflate its own
recurrence score.
Fires on post-verdict fire-and-forget (can't break the audit if
disk write fails). The learning loop is now closed: each audit
contributes to the KB that guides the next audit.
Tested: unit tests for normalizedSignature confirmed path-agnostic
dedup; static.ts regression tests confirmed rubric strings no longer
trip BLOCK while real unquoted unimplemented!() still does.
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Wires the cohesion-plan Phase C link: the scrum-master pipeline writes
per-file reviews to data/_kb/scrum_reviews.jsonl on accept; the
auditor now reads that same file and emits one kb_query finding per
scrum review whose `file` matches a path in the PR's diff.
Severity heuristic: attempt 1-3 → info, attempt 4+ → warn. Reaching
the cloud specialist (attempt 4+) means the ladder had to escalate,
which is meaningful signal reviewers should see. Tree-split fired is
also surfaced in the finding summary.
audit.ts now passes pr.files.map(f => f.path) into runKbCheck (the
old signature dropped it on the floor). Also adds auditor/audit_one.ts
— a dry-run CLI for auditing a single PR without posting to Gitea,
useful for verifying check behavior without spamming review comments.
Verified: after writing scrum_reviews for auditor/audit.ts and
mcp-server/observer.ts (both in PR #7), audit_one 7 surfaced both as
info findings with preview + accepted_model + tree_split flag. A
scrum review for playbook_memory.rs (NOT in PR #7) was correctly
filtered out.
lakehouse/auditor all checks passed (4 findings, all info)
auditor/index.ts (task #9) — the top-level poller. 90s interval,
dedupes by head SHA via data/_auditor/state.json, supports --once
for CLI testing. Env gates: LH_AUDITOR_RUN_DYNAMIC=1 to include
the hybrid fixture (default off; it mutates live state),
LH_AUDITOR_SKIP_INFERENCE=1 for fast runs without cloud calls.
Single-shot run proof (task #10):
cycle 1: 2 open PRs
audit PR #2 f0a3ed68 "Fix: UpsertOutcome newtype serde panic"
verdict=block, 9 findings (1 block, 5 warn, 3 info)
audit PR #1 039ed324 "Auditor: PR-claim hard-block reviewer"
verdict=approve, 4 findings (0 block, 0 warn, 4 info)
audits_run=2, state persisted
Commit statuses and issue comments posted live to Gitea. PR #2 is
currently hard-blocked (lakehouse/auditor commit status = failure);
PR #1 has a passing status. State survives restart — next cycle
skips already-audited SHAs.
Both PRs now have the audit comment with per-check breakdown.
Operator can read the comment, fix blocking findings (or defend
them with a reply), push a new commit; auditor re-audits on new
SHA, verdict updates, merge gate responds accordingly.
The full loop J asked for is closed:
1. static check caught own Phase 45 placeholder (b933334)
2. hybrid fixture caught UpsertOutcome serde panic (9c893fb)
3. LLM-Team-style codereview caught ternary bug (5bbcaf4)
4. auditor poller now runs on every open PR, block/approve with
evidence, re-audits on new SHAs
Tasks done: 1-11 (except 12, a scoped follow-up fix for UPDATE
branch dropping doc_refs). The auditor is running, catching real
bugs in its own build, and gating merges.
After the serde fix (PR #2, fix/upsert-outcome-serde) landed on main,
re-running this fixture STILL reported "doc_refs field is empty" —
but with a different root cause than the panic.
Root cause: pre-fix runs panicked on response serialization but had
already added entries to state (panic happened between upsert_entry
returning and the handler's serde_json::json! of the response). So
state.json was polluted with __auditor_test_worker__ entries from
those runs, WITHOUT doc_refs (doc_refs wasn't even wired at the time
those state rows were written).
The fixture's `find(endorsed_names.includes(TEST_WORKER_NAME))` was
picking the oldest polluted entry, not the fresh one.
Compounding: discovered a secondary bug while investigating —
upsert_entry's UPDATE branch only merges endorsed_names. doc_refs,
schema_fingerprint, valid_until on an UPDATE are silently dropped.
Filed as task #12, separate PR to follow.
Fix in this fixture: use a nonce suffix on both TEST_WORKER_NAME and
TEST_OPERATION so every run is guaranteed to hit the ADD path in
upsert_entry, sidestepping the UPDATE bug AND eliminating state
pollution entirely.
Live re-run after this edit:
✓ Phase 38 /v1/chat 449ms, 42 tokens
✓ Phase 40 Langfuse trace 20ms
✓ Phase 45.1 seed + doc_refs 239ms, doc_refs.length=1 persisted
✓ Phase 45.2 bridge diff 2ms, drifted=true
✗ Phase 45.3 drift-check HONEST 404 (endpoint not built)
shipped_phases: [38, 40, 45.1, 45.2] (was [38, 40, 45.2])
placeholder: [45.3] (was [45.1, 45.3])
One fewer placeholder — exactly because the serde fix merged on
fix/upsert-outcome-serde and the fixture now cleanly exercises the
path. The loop is:
fixture finds bug → PR fixes bug → fixture re-run confirms fix →
one fewer placeholder.
Caught by running a side-test through LLM Team's run_codereview
flow (gpt-oss:120b reviewer) against this fixture, 2026-04-22.
BEFORE:
const ourStart = Date.parse(
l1.evidence.match(/tokens=/) ? result.ran_at : result.ran_at
);
// Both branches return result.ran_at — the ternary is meaningless.
// result.ran_at is the fixture start time, NOT the moment we fired
// /v1/chat. Any trace created between fixture-start and chat-fetch
// would false-negative.
AFTER:
const chat_request_sent_ms = Date.now(); // captured before layer 1
// ...
const recent = items.filter(t =>
Date.parse(t.timestamp) >= chat_request_sent_ms
);
Re-ran the fixture against the live stack — layers 1,2,4 still pass
(no regression); layer 2 trace matched at age=2494ms which is within
the chat-to-trace propagation window. Layers 3,5 still fail for the
original unrelated reasons (UpsertOutcome serde panic + Phase 45
slice 3 endpoint not built).
First concrete act-on-finding from a code-checker run. The process
works.
auditor/fixtures/hybrid_38_40_45.ts — the never-before-run hybrid
test. Exercises Phase 38 /v1/chat → Phase 40 Langfuse → Phase 45
slice 1 seed+doc_refs → Phase 45 slice 2 bridge drift → (expected-
fail) Phase 45 slice 3 drift-check endpoint.
auditor/fixtures/cli.ts — standalone runner. Human-readable summary
to stderr, machine-readable JSON to stdout, exit code 0/1/2 for
pass / fail / partial_pass.
Live run results — honest measurements, not hand-waved:
✓ Phase 38 /v1/chat returns 9 visible tokens, 6.7s latency
("docker run is a common Docker command.")
✓ Phase 40 Langfuse trace 18a8a0b7 landed in 2.5s
✗ Phase 45.1 seed endpoint returns empty reply — discovered a
PRE-EXISTING BUG unrelated to doc_refs:
playbook_memory.rs:257 UpsertOutcome has newtype
variants Added(String) and Noop(String) under
#[serde(tag="mode")] — serde panics on serialize.
panicked at crates/vectord/src/service.rs:2323:
Error("cannot serialize tagged newtype variant
UpsertOutcome::Added containing a string")
Reproduced: curl /seed with AND without doc_refs
both get "Empty reply from server" (socket closed
mid-response). This bug has existed since Phase 26
shipped (commit 640db8c, 2026-04-21). No test or
caller in the repo exercised the response path live
against the gateway until this fixture did.
✓ Phase 45.2 context7 bridge confirms drift: current hash
475a0396ca436bba vs our stale input, upstream last
updated 2026-04-20
✗ Phase 45.3 /doc_drift/check endpoint — correctly unreachable
because layer 3 blocked us from getting a playbook_id;
endpoint still doesn't exist independent of that
Real numbers published: per-layer latency_ms, token counts,
trace_age_ms, library_id, current_hash_length. All stored in the
JSON output for downstream audit.
Value delivered: the fixture's first live run found a bug that
unit tests, compile checks, and my own "phase shipped" commits all
missed. Exactly the gap J called out — the auditor is doing what
it's supposed to do.
Bug fix is a SEPARATE concern: new task #11 tracks a separate PR
(fix/upsert-outcome-serde) so the audit finding and the fix stay
cleanly attributed.
auditor/checks/static.ts — grep-style scan of PR diffs, no AST,
no LLM. High-signal patterns only.
Severity grading:
- BLOCK — unimplemented!(), todo!(), panic!("not implemented"),
throw new Error("not implemented")
- WARN — TODO/FIXME/XXX/HACK in added lines;
new pub struct fields with <2 mentions in the diff
(added but nobody reads it — placeholder state)
- INFO — hardcoded "placeholder"/"dummy"/"foobar"/"changeme"/"xxx"
strings in added lines
Live-proven — the existential test J asked for:
vs PR #1 (scaffold): 0 findings (all scaffold fields cross-
reference within the diff)
vs commit 2a4b81b (Phase 5 WARN: every DocRef field (tool,
45 first slice — I version_seen, snippet_hash, source_url,
half-admitted placeholder): seen_at) added with 0 read-sites in
the diff
That's the auditor flagging my own "Phase 45 first slice" commit as
state-without-consumer, which is exactly what I half-admitted it
was. If PR #1 had required auditor-pass (branch protection), the
DocRef commit would have been blocked pre-merge. The auditor works
because it agreed with the honest read.
Next: dynamic hybrid test fixture (task #4) — the never-run multi-
layer pipeline test.
auditor/claim_parser.ts — reads PR body + commit messages, extracts
ship-claims. Regex-based, intentionally not LLM-driven: the parser's
job is to surface claim substrates, not to judge them (that's the
inference check's job, runs later with cloud model).
Three strength tiers:
- strong — "verified end-to-end", "live-proven", "production-ready",
"phase N shipped", "proven"
- moderate — "shipped", "landed", "green", "passing", "works",
"complete", "done"
- weak — "should work", "expected to", "probably"
Live-proven against PR #1 (this PR): 4 claims extracted from
1 commit (2 strong, 2 moderate). "live-proven" correctly tagged as
strong (it IS a stronger claim than "shipped").
Next: static diff check consumes these claims + the PR diff to find
placeholder patterns — empty fns, TODO, unwired fields, etc.
All-Bun sub-agent that watches open PRs on Gitea, reads ship-claims,
and hard-blocks merges when the code doesn't back the claim. First
commit of N; this is the skeleton. Dynamic/static/inference/kb checks
+ poller land in follow-up commits on this same branch.
- auditor/types.ts — Claim, Finding, Verdict, PrSnapshot shapes
- auditor/gitea.ts — minimal API client (listOpenPrs, getPrDiff,
postCommitStatus, postReview). Live-proven: returned 0 open PRs
against our repo (which IS the current state — every commit today
went to main directly, which is the problem this auditor is meant
to prevent)
- auditor/policy.ts — stub `assembleVerdict` + severity rules.
Intentionally conservative defaults: strong claim + zero evidence
= block, not warn.
- auditor/README.md — how to run + the hard-block mechanism
Workflow discipline change: starting with this branch, no more
direct pushes to main. Every change lands as a PR. When this
auditor is fully built and running, it'll review its own
completion PR — the recursive self-test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
