3 Commits

Author SHA1 Message Date
profit
0306dd88c1 auditor: close the verdict→playbook loop + fix rubric-string false positive
Some checks failed
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Two changes that fell out of running the auto-loop for real on PR #8:

1. The systemd auditor blocked PR #8 on 'unimplemented!()' / 'todo!()'
   in tests/real-world/hard_task_escalation.ts — but those strings are
   the rubric itself, not macro calls. Added isInsideQuotedString()
   detection in static.ts: BLOCK_PATTERNS now skip matches that fall
   inside double-quoted / single-quoted / backtick string literals on
   the added line. WARN/INFO patterns still run — a TODO comment in
   a string is still a valid signal.

2. Verdicts were being persisted to disk but never fed back as
   learning signal. Added appendAuditLessons() — every block/warn
   finding writes a JSONL row to data/_kb/audit_lessons.jsonl with a
   path-agnostic signature (strips file paths, line numbers, commit
   hashes) so the SAME class of finding on DIFFERENT files dedups to
   one signature.

   kb_query now tails audit_lessons.jsonl and emits recurrence
   findings: 2 distinct PRs hit a signature = info, 3-4 = warn, 5+ =
   block. Severity ramps on distinct-PR count, not total rows, so a
   single unfixed PR being re-audited doesn't inflate its own
   recurrence score.

Fires on post-verdict fire-and-forget (can't break the audit if
disk write fails). The learning loop is now closed: each audit
contributes to the KB that guides the next audit.

Tested: unit tests for normalizedSignature confirmed path-agnostic
dedup; static.ts regression tests confirmed rubric strings no longer
trip BLOCK while real unquoted unimplemented!() still does.
2026-04-22 21:31:35 -05:00
profit
dc01ba0a3b auditor: kb_query surfaces scrum-master reviews for files in PR diff
Some checks failed
lakehouse/auditor 2 blocking issues: unimplemented!() macro call in tests/real-world/hard_task_escalation.ts
Wires the cohesion-plan Phase C link: the scrum-master pipeline writes
per-file reviews to data/_kb/scrum_reviews.jsonl on accept; the
auditor now reads that same file and emits one kb_query finding per
scrum review whose `file` matches a path in the PR's diff.

Severity heuristic: attempt 1-3 → info, attempt 4+ → warn. Reaching
the cloud specialist (attempt 4+) means the ladder had to escalate,
which is meaningful signal reviewers should see. Tree-split fired is
also surfaced in the finding summary.

audit.ts now passes pr.files.map(f => f.path) into runKbCheck (the
old signature dropped it on the floor). Also adds auditor/audit_one.ts
— a dry-run CLI for auditing a single PR without posting to Gitea,
useful for verifying check behavior without spamming review comments.

Verified: after writing scrum_reviews for auditor/audit.ts and
mcp-server/observer.ts (both in PR #7), audit_one 7 surfaced both as
info findings with preview + accepted_model + tree_split flag. A
scrum review for playbook_memory.rs (NOT in PR #7) was correctly
filtered out.
2026-04-22 21:18:21 -05:00
profit
039ed32411 Auditor: KB query check + verdict orchestrator + Gitea poster
All checks were successful
lakehouse/auditor all checks passed (4 findings, all info)
auditor/checks/kb_query.ts (task #7) — reads data/_kb/outcomes.jsonl,
error_corrections.jsonl, data/_observer/ops.jsonl, data/_bot/cycles/*.
Cheap/offline: no model calls, tail-reads only. Fail-rate >30% in
recent scenario outcomes → warn; otherwise info. Live-proven: 1
finding emitted against current KB state (69 scenario runs, 27.7%
fail rate — below warn threshold).

auditor/audit.ts (task #8) — orchestrator. Runs static + dynamic +
inference + kb_query in parallel, calls assembleVerdict, persists
to data/_auditor/verdicts/, posts to Gitea (commit status + issue
comment). AuditOptions supports skip_dynamic/skip_inference/dry_run
for iteration.

auditor/gitea.ts — added postIssueComment (author can comment on
own PR, unlike postReview which self-review-blocks).

static.ts — skip BLOCK_PATTERNS scan on auditor/checks/* and
auditor/fixtures/* because those files legitimately contain the
patterns as regex/string-literal data. WARN/INFO patterns (TODO
comments, hardcoded placeholders) still run. Live-proven: dry-run
audit of PR #1 after fix went from 13 block findings to 0 from
static; 11 warn from inference still fire on real overreach claims.

Dry-run audit against PR #1, skip_dynamic=true:
  verdict: block (BEFORE the static fix)
  verdict: request_changes (AFTER — inference correctly flagged
           "tasks 1-9 complete" as not backed; 0 false-positive
           blocks from static self-match)
  42.5s total across checks (mostly cloud inference: 36s)
  26 claims, 39KB diff

Tasks 5 + 6 + 7 + 8 complete. Remaining: #9 (poller) + #10
(end-to-end proof) + #12 (upsert UPDATE merge fix).
2026-04-22 03:59:38 -05:00