lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Observed on PR #8 audit (de11ac4): 7 warn findings, all from the
cloud inference check. Investigation showed two distinct bug classes
that weren't "ship bad code", they were "auditor misreads the diff":
1. Cloud flagged "X not defined in this diff / missing implementation"
for symbols like `tailJsonl` and `stubFinding` that ARE defined —
just not in the added lines of this diff. Fix: extract candidate
symbols from the cloud's gap summary, grep the repo for their
definitions (function/const/let/def/class/struct/enum/trait/fn).
If every named symbol resolves, drop the finding; if some do,
demote to info with the resolution in evidence.
2. Cloud flagged runtime metrics like "58 cloud calls, 306s
end-to-end" as unbacked claims. These are empirical outputs
from running the test, not things a static diff can prove.
Fix: claim_parser now has an `empirical` strength class
matching iteration counts, cloud-call counts, duration metrics,
attempt counts, tier-count phrases. Inference drops empirical
claims from its cloud prompt (verifiable[] subset only) and
claim-index mapping uses verifiable[] so cloud responses still
line up.
Added `claims_empirical` to audit metrics so the verdict is
introspectable: how many claims WERE runtime-only vs how many
are diff-verifiable?
Verified: unit tests confirm empirical classification on 5
sample commit messages; symbol resolver found both false-positive
symbols (tailJsonl + stubFinding) and correctly skipped a known-
fake symbol.
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Two changes that fell out of running the auto-loop for real on PR #8:
1. The systemd auditor blocked PR #8 on 'unimplemented!()' / 'todo!()'
in tests/real-world/hard_task_escalation.ts — but those strings are
the rubric itself, not macro calls. Added isInsideQuotedString()
detection in static.ts: BLOCK_PATTERNS now skip matches that fall
inside double-quoted / single-quoted / backtick string literals on
the added line. WARN/INFO patterns still run — a TODO comment in
a string is still a valid signal.
2. Verdicts were being persisted to disk but never fed back as
learning signal. Added appendAuditLessons() — every block/warn
finding writes a JSONL row to data/_kb/audit_lessons.jsonl with a
path-agnostic signature (strips file paths, line numbers, commit
hashes) so the SAME class of finding on DIFFERENT files dedups to
one signature.
kb_query now tails audit_lessons.jsonl and emits recurrence
findings: 2 distinct PRs hit a signature = info, 3-4 = warn, 5+ =
block. Severity ramps on distinct-PR count, not total rows, so a
single unfixed PR being re-audited doesn't inflate its own
recurrence score.
Fires on post-verdict fire-and-forget (can't break the audit if
disk write fails). The learning loop is now closed: each audit
contributes to the KB that guides the next audit.
Tested: unit tests for normalizedSignature confirmed path-agnostic
dedup; static.ts regression tests confirmed rubric strings no longer
trip BLOCK while real unquoted unimplemented!() still does.
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Wires the cohesion-plan Phase C link: the scrum-master pipeline writes
per-file reviews to data/_kb/scrum_reviews.jsonl on accept; the
auditor now reads that same file and emits one kb_query finding per
scrum review whose `file` matches a path in the PR's diff.
Severity heuristic: attempt 1-3 → info, attempt 4+ → warn. Reaching
the cloud specialist (attempt 4+) means the ladder had to escalate,
which is meaningful signal reviewers should see. Tree-split fired is
also surfaced in the finding summary.
audit.ts now passes pr.files.map(f => f.path) into runKbCheck (the
old signature dropped it on the floor). Also adds auditor/audit_one.ts
— a dry-run CLI for auditing a single PR without posting to Gitea,
useful for verifying check behavior without spamming review comments.
Verified: after writing scrum_reviews for auditor/audit.ts and
mcp-server/observer.ts (both in PR #7), audit_one 7 surfaced both as
info findings with preview + accepted_model + tree_split flag. A
scrum review for playbook_memory.rs (NOT in PR #7) was correctly
filtered out.
