auditor: inference temp=0, think=false — kill signature creep
9-run empirical test showed 20 of 27 audit_lessons signatures were singletons (count=1) — the cloud producing slightly-different summary phrasings for the SAME underlying claim on each audit, each hashing to a fresh signature. That's the creep J flagged — not explosive, but steady ~2 new sigs per run, unbounded over hundreds of runs. Root cause: temperature=0.2 + think=true was letting variable prose leak into the classification output. Fix: temp=0 (greedy sample → identical input yields identical output on same model version), think=false (no reasoning trace variance), max_tokens 3000→1500 (tighter bound prevents tail wander). The compounding policy itself was validated by the 9 runs: - 7 recurring claims (the legitimate signals) all at conf 0.08-0.20 - ratingSeverity() correctly held them at info (below 0.3 threshold) - cross-PR signal test separately confirmed conf=1.00 → sev=block Also: LH_AUDIT_RUNS env so the test can validate with smaller N.
This commit is contained in:
parent
81a2200344
commit
1e00eb4472
@ -112,9 +112,19 @@ export async function runInferenceCheck(claims: Claim[], diff: string): Promise<
|
|||||||
{ role: "system", content: systemMsg },
|
{ role: "system", content: systemMsg },
|
||||||
{ role: "user", content: userMsg },
|
{ role: "user", content: userMsg },
|
||||||
],
|
],
|
||||||
max_tokens: 3000,
|
// Deterministic classification mode — temp=0 is greedy-sample,
|
||||||
temperature: 0.2,
|
// so identical input → identical output on the same model
|
||||||
think: true, // T3 overseer should reason — JSON shape is still required
|
// version. think=false disables the reasoning trace that was
|
||||||
|
// letting variable prose leak into the classification output
|
||||||
|
// and inflate the audit_lessons signature set (observed as
|
||||||
|
// sig_count creep across the 9-run empirical test).
|
||||||
|
//
|
||||||
|
// max_tokens tightened to 1500 — the structured JSON response
|
||||||
|
// fits comfortably in 1500 tokens for typical PRs (~7 claims);
|
||||||
|
// the old 3000 just gave the model room to wander.
|
||||||
|
max_tokens: 1500,
|
||||||
|
temperature: 0,
|
||||||
|
think: false,
|
||||||
}),
|
}),
|
||||||
signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
|
signal: AbortSignal.timeout(CALL_TIMEOUT_MS),
|
||||||
});
|
});
|
||||||
|
|||||||
@ -28,7 +28,7 @@ const AUDIT_LESSONS = `${REPO}/data/_kb/audit_lessons.jsonl`;
|
|||||||
const VERDICTS_DIR = `${REPO}/data/_auditor/verdicts`;
|
const VERDICTS_DIR = `${REPO}/data/_auditor/verdicts`;
|
||||||
const POLL_INTERVAL_MS = 5_000;
|
const POLL_INTERVAL_MS = 5_000;
|
||||||
const AUDIT_TIMEOUT_MS = 180_000;
|
const AUDIT_TIMEOUT_MS = 180_000;
|
||||||
const RUNS = 9;
|
const RUNS = Number(process.env.LH_AUDIT_RUNS ?? 9);
|
||||||
const TARGET_PR = Number(process.env.LH_AUDIT_PR ?? 8);
|
const TARGET_PR = Number(process.env.LH_AUDIT_PR ?? 8);
|
||||||
|
|
||||||
async function sh(cmd: string): Promise<{ stdout: string; stderr: string; code: number }> {
|
async function sh(cmd: string): Promise<{ stdout: string; stderr: string; code: number }> {
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user