Pre-2026-04-25 the scrum_master applied a hardcoded grounding-rate gate
inline. That baked policy into the wrong layer — semantic judgment about
whether a review is grounded belongs in the observer (which has Langfuse
traces, sees every response across the system, and can call cloud LLMs
for real evaluation). Scrum should report DATA, observer DECIDES.
What landed:
- scrum_master_pipeline.ts: removed the inline grounding-pct threshold;
every accepted candidate now POSTs to observer's /review endpoint with
{response, source_content, grounding_stats, model, attempt}. Observer
returns {verdict: accept|reject|cycle, confidence, notes}. On observer
failure, scrum falls open to accept (observer is policy, not blocker).
- mcp-server/observer.ts: new POST /review endpoint with two-tier
evaluator. Tier 1: cloud LLM (qwen3-coder:480b at temp=0) hand-reviews
with full context — response + source excerpt + grounding stats — and
emits structured verdict JSON. Tier 2: deterministic heuristic over
grounding pct + total quotes when cloud throttles, marked source:
"heuristic" so consumers can tune it later by comparing against cloud.
- Every verdict persists to data/_kb/observer_reviews.jsonl with full
input snapshot so cloud vs heuristic can be A/B compared once cloud
quota refreshes.
Verified end-to-end: smoke loop iter 1 — observer returned `cycle` on
21% grounding (cycled to next rung), `reject` on 17% (gave up). Iter 2
— `reject` on 12% and 14%. Both UNRESOLVED with honest signal instead
of polluting pathway memory with hallucinated patterns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3f166a5558