f4be27a879
Some checks failed
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Observed on PR #8 audit (de11ac4): 7 warn findings, all from the cloud inference check. Investigation showed two distinct bug classes that weren't "ship bad code", they were "auditor misreads the diff": 1. Cloud flagged "X not defined in this diff / missing implementation" for symbols like `tailJsonl` and `stubFinding` that ARE defined — just not in the added lines of this diff. Fix: extract candidate symbols from the cloud's gap summary, grep the repo for their definitions (function/const/let/def/class/struct/enum/trait/fn). If every named symbol resolves, drop the finding; if some do, demote to info with the resolution in evidence. 2. Cloud flagged runtime metrics like "58 cloud calls, 306s end-to-end" as unbacked claims. These are empirical outputs from running the test, not things a static diff can prove. Fix: claim_parser now has an `empirical` strength class matching iteration counts, cloud-call counts, duration metrics, attempt counts, tier-count phrases. Inference drops empirical claims from its cloud prompt (verifiable[] subset only) and claim-index mapping uses verifiable[] so cloud responses still line up. Added `claims_empirical` to audit metrics so the verdict is introspectable: how many claims WERE runtime-only vs how many are diff-verifiable? Verified: unit tests confirm empirical classification on 5 sample commit messages; symbol resolver found both false-positive symbols (tailJsonl + stubFinding) and correctly skipped a known- fake symbol.
73 lines
2.8 KiB
TypeScript
73 lines
2.8 KiB
TypeScript
// Shared types for the claim-auditor. Every field exists for a reason;
|
|
// if something can't be verified from a check, it goes into `evidence`
|
|
// so the verdict is inspectable, not a black box.
|
|
|
|
export type CheckKind = "static" | "dynamic" | "inference" | "kb_query";
|
|
|
|
export type Severity = "info" | "warn" | "block";
|
|
|
|
export interface Claim {
|
|
// Verbatim phrase that raised the claim — e.g. "Phase 38 shipped",
|
|
// "verified end-to-end", "works after restart". Used as the "what
|
|
// does the author assert" input to downstream checks.
|
|
text: string;
|
|
// Where it came from. `commit_sha` is the short hash; `location`
|
|
// is a file:line for in-diff claims, or "pr_body" / "commit_message".
|
|
commit_sha: string;
|
|
location: string;
|
|
// Heuristic rating of how strong the claim is. "green+tested"
|
|
// is strong; "should work" is weak. Drives sensitivity — stronger
|
|
// claims get harder-blocked on weak evidence.
|
|
//
|
|
// "empirical" is a separate class: runtime measurements like
|
|
// "N cloud calls" / "306s end-to-end" / "accepted on attempt N".
|
|
// These cannot be verified from a static diff — only from the test
|
|
// output that produced them. Inference skips diff-verification for
|
|
// empirical claims; they become info-level context unless a future
|
|
// runtime_evidence check contradicts them.
|
|
strength: "weak" | "moderate" | "strong" | "empirical";
|
|
}
|
|
|
|
export interface Finding {
|
|
check: CheckKind;
|
|
severity: Severity;
|
|
claim_text?: string;
|
|
// Free-form short description: "field added but never read", "no
|
|
// test covers this code path", "cloud model says placeholder".
|
|
summary: string;
|
|
// Concrete evidence: file paths, line numbers, log excerpts, test
|
|
// output, cloud-model verdict. No handwaving.
|
|
evidence: string[];
|
|
}
|
|
|
|
export interface Verdict {
|
|
pr_number: number;
|
|
head_sha: string;
|
|
audited_at: string;
|
|
overall: "approve" | "request_changes" | "block";
|
|
findings: Finding[];
|
|
// Real numbers that downstream policy can gate on. e.g. if the
|
|
// hybrid test produced latency numbers or token counts, they
|
|
// surface here so /auditor/history is queryable.
|
|
metrics: Record<string, number>;
|
|
// Short one-line justification for the `overall` verdict. What
|
|
// gets posted as the commit-status description in Gitea (max 140
|
|
// chars) must fit here.
|
|
one_liner: string;
|
|
}
|
|
|
|
export interface PrSnapshot {
|
|
number: number;
|
|
head_sha: string;
|
|
base_sha: string;
|
|
title: string;
|
|
body: string;
|
|
state: "open" | "closed" | "merged";
|
|
author: string;
|
|
// Array of commit messages in the PR (not diffs — those are
|
|
// fetched on-demand per-check).
|
|
commits: Array<{ sha: string; message: string; author: string }>;
|
|
// File paths touched by the PR, with lines-added / lines-removed.
|
|
files: Array<{ path: string; additions: number; deletions: number }>;
|
|
}
|