16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
root
|
3f166a5558 |
scrum + observer: hand-review wire — judgment moved out of the inner loop
Pre-2026-04-25 the scrum_master applied a hardcoded grounding-rate gate
inline. That baked policy into the wrong layer — semantic judgment about
whether a review is grounded belongs in the observer (which has Langfuse
traces, sees every response across the system, and can call cloud LLMs
for real evaluation). Scrum should report DATA, observer DECIDES.
What landed:
- scrum_master_pipeline.ts: removed the inline grounding-pct threshold;
every accepted candidate now POSTs to observer's /review endpoint with
{response, source_content, grounding_stats, model, attempt}. Observer
returns {verdict: accept|reject|cycle, confidence, notes}. On observer
failure, scrum falls open to accept (observer is policy, not blocker).
- mcp-server/observer.ts: new POST /review endpoint with two-tier
evaluator. Tier 1: cloud LLM (qwen3-coder:480b at temp=0) hand-reviews
with full context — response + source excerpt + grounding stats — and
emits structured verdict JSON. Tier 2: deterministic heuristic over
grounding pct + total quotes when cloud throttles, marked source:
"heuristic" so consumers can tune it later by comparing against cloud.
- Every verdict persists to data/_kb/observer_reviews.jsonl with full
input snapshot so cloud vs heuristic can be A/B compared once cloud
quota refreshes.
Verified end-to-end: smoke loop iter 1 — observer returned `cycle` on
21% grounding (cycled to next rung), `reject` on 17% (gave up). Iter 2
— `reject` on 12% and 14%. Both UNRESOLVED with honest signal instead
of polluting pathway memory with hallucinated patterns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
c90a509f49 |
applier: LH_APPLIER_FILES env to constrain to current-iter targets
Without this, the applier loaded the latest 34 reviews and patched the highest-confidence file from history — which is meaningless when called from the autonomous loop where the intent is "review file X this iter, patch file X this iter." Now the loop passes its targets through and the applier filters eligible reviews accordingly. Causality is restored: scrum reviews file X → applier patches file X. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
9ecc5848fa |
scrum: blind-response guard + anchor-grounding post-verifier
Two signal-quality fixes for the scrum loop: 1. isBlindResponse() — detects models that emit structurally-valid review JSON containing "no source code visible / cannot verify" even when source WAS supplied. Rejects so the ladder cycles to the next rung instead of accepting the blind hallucination. 2. verifyAnchorGrounding() + appendGroundingFooter() — post-process verifier that extracts every backtick-quoted snippet from the review and checks it against the original source content. Appends a grounding footer reporting grounded vs ungrounded counts so humans can audit hallucination rate at a glance. Born from the iter where llm_team_ui.py review came back with 6/10 findings hallucinated (invented render_template_string calls, fabricated logger.exception sites, made-up SHA-256 hashing). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
ed85620558 |
scrum: filter table-header words from bug_fingerprint extraction
Iter 11 surfaced "DeadCode:Flag" in the matrix — a noisy pattern_key where "Flag" is the table column HEADER kimi produces for structured review output, not an actual Rust identifier. Kimi's standard format on recent iters: | # | Change | Flag | Confidence | | 1 | Wire AgentIdentity into.. | Boundary.. | 92% | The extractor's KEYWORDS set already filtered Rust grammar words (self, mut, async, etc) and the FLAG_VARIANTS themselves. Adding markdown-layout words (Flag, Change, Confidence, PRD, Plan) closes the last common noise class. One-line addition — empirically validated against the iter 11 vectord trace that produced DeadCode:Flag. Future iters won't reproduce that specific noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
0cf1b7c45a |
scrum_master: env-configurable tree-split threshold + shard size
Some checks failed
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Hard-coded constants (FILE_TREE_SPLIT_THRESHOLD=6000, FILE_SHARD_SIZE=3500) were tuned for Rust source files in crates/<crate>/src/*.rs. Running the pipeline against /root/llm-team-ui/llm_team_ui.py (13K lines, ~400KB) would produce ~200 shards per review at the default size — not viable. Two env vars now: - LH_SCRUM_TREE_SPLIT_THRESHOLD — when tree-split fires (default 6000) - LH_SCRUM_SHARD_SIZE — bytes per shard (default 3500) For the big-Python case the CLAUDE.md in /root/llm-team-ui/ recommends LH_SCRUM_TREE_SPLIT_THRESHOLD=20000, LH_SCRUM_SHARD_SIZE=12000 which brings the 13K-line file down to ~35 shards — same ballpark as a typical Rust file review. No default change. Existing lakehouse runs unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
f4cff660aa |
ADR-021 Phase D fix: strip flag names + Rust keywords from pattern_keys
Iter 9 revealed two quality bugs in the extractor: 1. Kimi wraps the Flag column in backticks (\`DeadCode\`), so the flag name itself was captured as a code token. Result: pattern_keys like "DeadCode:DeadCode" that match nothing and add noise to the index. Fix: filter FLAG_VARIANTS out of token candidates. 2. Complex backtick content like \`Foo::bar(&self) -> u64\` was rejected wholesale by the identifier regex. Fallback now scans for identifier substrings and ranks by ::-qualified paths first, then length. Bonus: filter Rust keywords (self, mut, async, etc) since they're grammar, not bug-shape signal. Dry-run on iter 9 delta.rs output produces semantically meaningful keys: DeadCode:DeltaStats::tombstones_applied NullableConfusion:DeltaError-DeltaStats-apply_delta BoundaryViolation:apply_delta-journald::emit-rows_dropped_by_tombstones PseudoImpl:apply_delta-delta_ops-validate_schema These are stable under reviewer prose variation (canonical sort + top-3 slice) and precise enough to separate different bugs within the same Flag category. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
ee31424d0c |
ADR-021 Phase D: bug_fingerprint pattern extraction from reviewer output
Some checks failed
lakehouse/auditor 4 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Fills the gap between Phase B (flags tagged) and Phase C (preamble
quotes past fingerprints): parses each reviewer line that mentions a
Flag variant, collects backtick-quoted identifiers, canonicalizes them
(sorted alphabetically, top 3), and emits a stable pattern_key of
shape `{Flag}:{tok1}-{tok2}-{tok3}`.
Stability by design: canonical sort means "row_count + QueryResponse"
and "QueryResponse + row_count" produce the same key, so variation in
reviewer prose doesn't fragment the index. Top-3 cap keeps keys short
while retaining enough signal to separate different bugs of the same
category.
Dry-run validation on iter-8 delta.rs output (crates/queryd prefix)
extracted 10 semantically meaningful fingerprints including:
- UnitMismatch:base_rows-checked_add-checked_sub
- DeadCode:queryd::delta::write_delta (P9-001 dead-function finding)
- BoundaryViolation:can_access-log_query-masked_columns (P13-001 gap)
- NullableConfusion:CompactResult-DeltaError-IntegerOverflow
Cross-cutting signal: kimi-k2:1t's finding #5 explicitly quoted the
seeded pathway memory preamble ("Pathway memory flags row_count-
file_count unit mismatch") and proposed overflow-checked arithmetic as
the fix. That is the compounding loop in action — prior bug context
shifted the reviewer's attention toward a specific instance of the
same class, which produces a specific pattern_key that will compound
further on the next iter.
Filter: identifier-shaped tokens only (A-Za-z_ / :: paths / snake_case
/ CamelCase). Skips punctuation, prose quotes, and tokens <3 chars so
generic nouns and partial words don't pollute the index.
What's still queued (Phase E):
- type_hints_used population from catalogd column types + Arrow schema
- auditor → pathway audit_consensus update wire (strict-audit gate
activation)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
0a0843b605 |
ADR-021: semantic-correctness layer lands in pathway_memory (A+B+C)
Some checks failed
lakehouse/auditor 4 blocking issues: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
Phase A — data model (vectord/src/pathway_memory.rs):
+ SemanticFlag enum (9 variants: UnitMismatch, TypeConfusion,
NullableConfusion, OffByOne, StaleReference, PseudoImpl, DeadCode,
WarningNoise, BoundaryViolation) as #[serde(tag = "kind")]
+ TypeHint { source, symbol, type_repr }
+ BugFingerprint { flag, pattern_key, example, occurrences }
+ PathwayTrace gains semantic_flags, type_hints_used, bug_fingerprints
all #[serde(default)] for back-compat deserialization of pre-ADR-021
traces on disk
+ build_pathway_vec now tokenizes flag:{variant} + bug:{flag}:{key}
so traces with different bug histories cluster separately in the
similarity gate (proven by pathway_vec_differs_when_bug_fingerprint_added
test)
Phase B — producer (scrum_master_pipeline.ts):
+ Prompt addendum: each finding must carry `**Flag: <CATEGORY>**` tag
alongside the existing Confidence: NN% tag. 9 category choices plus
`None` for improvements that aren't bug-shaped.
+ Parser extracts tagged flags from reviewer markdown; falls back to
bare-word match if reviewer omits the label. Deduplicated per trace.
+ PathwayTracePayload gains semantic_flags / type_hints_used /
bug_fingerprints fields. Wire format matches Rust serde tagged enum
so TS and Rust interop directly.
Phase C — pre-review enrichment:
+ new `/vectors/pathway/bug_fingerprints` endpoint aggregates
occurrences by (flag, pattern_key) across traces sharing a narrow
fingerprint, sorts by frequency, returns top-K.
+ scrum calls it before the ladder and prepends a PATHWAY MEMORY
preamble to the reviewer prompt ("these patterns appeared N times
on this file area before — check for recurrences"). Empty on
fresh install; grows as the matrix index learns.
Tests: 27 pathway_memory tests green (was 18). New tests:
- pathway_trace_deserializes_without_new_fields_backcompat
- semantic_flag_serializes_as_tagged_enum
- bug_fingerprint_roundtrips_through_serde
- pathway_vec_differs_when_bug_fingerprint_added
- semantic_flag_discriminates_by_variant
- bug_fingerprints_aggregate_by_pattern_key (sums occurrences, sorts desc)
- bug_fingerprints_empty_for_unseen_fingerprint
- bug_fingerprints_respects_limit
- insert_preserves_semantic_fields (roundtrip via persist + reload)
Workspace warnings unchanged at 11.
What's still queued (not this commit):
- type_hints_used population from catalogd column types + Arrow schema
- bug_fingerprint extraction from reviewer output (Phase D — for now
semantic_flags populate but the fingerprint key requires parsing
code-shape from the finding; next iteration's work)
- auditor → pathway audit_consensus update wire (explicit-fail gate)
Why this commit matters: the mechanical applier's gates are syntactic
(warning count, patch size, rationale-token alignment). The
queryd/delta.rs base_rows bug (86901f8) was found by human reading —
unit mismatch between row counts and file counts. At 100 bugs this
deep, humans can't catch them all; the matrix index has to learn the
shapes. This commit gives it the fields to learn into and the surface
to read from.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
2f8b347f37 |
pathway_memory: consensus-designed sidecar + hot-swap learning loop
Some checks failed
lakehouse/auditor 11 warnings — see review
10-probe N=3 consensus (kimi-k2:1t / gpt-oss:120b / qwen3.5:latest /
deepseek-v3.1:671b / qwen3-coder:480b / mistral-large-3:675b /
qwen3.5:397b + 2 stability re-probes; 2 openrouter probes 429'd) locked
the design across three rounds. Full JSON responses in
data/_kb/consensus_reducer_design_{mocq3akn,mocq6pi1,mocqatik}.json.
What it does
Preserves FULL backtrack context per reviewed file (ladder attempts +
latencies + reject reasons, KB chunks with provenance + cosine + rank,
observer signals, context7 bridge hits, sub-pipeline calls, audit
consensus) and indexes them by narrow fingerprint for hot-swap of
proven review pathways.
When scrum reviews a file:
1. narrow fingerprint = task_class + file_prefix + signal_class
2. query_hot_swap checks pathway memory for a match that passes
probation (≥3 replays @ ≥80% success) + audit gate + similarity
(≥0.90 cosine on normalized-metadata-token embedding)
3. if hot-swap eligible, recommended model tried first in the ladder
4. replay outcome reported back, updating the pathway's success_rate
5. pathways below 0.80 after ≥3 replays retire permanently (sticky)
6. full PathwayTrace always inserted at end of review — hot-swap
grows with use, it doesn't bootstrap from nothing
Gate design is load-bearing:
- narrow fingerprint (6 of 8 consensus models converged on the same
3-field composition; lock) — enables generalization within crate
- probation ≥3 replays — binomial tail at 80% is ~5%, below is noise
- success rate ≥0.80 — mistral + qwen3-coder independently proposed
this exact threshold across two rounds
- similarity ≥0.90 — middle of the 0.85/0.95 consensus spread
- bootstrap: null audit_consensus ALLOWED (auditor → pathway update
not wired yet; probation + success_rate gates alone enforce safety
during bootstrap; explicit audit FAIL still blocks)
- retirement is sticky — prevents oscillation on noise
Files
+ crates/vectord/src/pathway_memory.rs (new, 600 lines + 18 tests)
PathwayTrace, LadderAttempt, KbChunkRef, ObserverSignal, BridgeHit,
SubPipelineCall, AuditConsensus, HotSwapCandidate, PathwayMemory,
PathwayMemoryStats. 18/18 tests green.
Cosine + 32-bucket L2-normalized embedding; mirror of TS impl.
M crates/vectord/src/lib.rs
pub mod pathway_memory;
M crates/vectord/src/service.rs
VectorState grows pathway_memory field;
4 HTTP handlers (/pathway/insert, /pathway/query,
/pathway/record_replay, /pathway/stats).
M crates/gateway/src/main.rs
Construct PathwayMemory + load from storage on boot,
wire into VectorState.
M tests/real-world/scrum_master_pipeline.ts
Byte-matching TS bucket-hash (verified same bucket indices as
Rust); pre-ladder hot-swap query; ladder reorder on hit;
per-attempt latency capture; post-accept trace insert
(fire-and-forget); replay outcome recording;
observer /event emits pathway_hot_swap_hit, pathway_similarity,
rungs_saved per review for the VCP UI.
M ui/server.ts
/data/pathway_stats aggregates /vectors/pathway/stats +
scrum_reviews.jsonl window for the value metric.
M ui/ui.js
Three new metric cards:
· pathway reuse rate (activity: is it firing?)
· avg rungs saved (value: is it earning its keep?)
· pathways tracked (stability: retirement = learning)
What's not in this commit (queued)
- auditor → pathway audit_consensus update wire (explicit audit-fail
block activates when this lands)
- bridge_hits + sub_pipeline_calls population from context7 / LLM
Team extract results (fields wired, callers not yet)
- replay log (PathwayReplayOutcome {matched_id, succeeded, ts}) as
a separate jsonl for forensic audit of why specific replays failed
Why > summarization
Summaries discard the causal chain. With this, auditor can verify
citation provenance, applier can distinguish lucky from learned paths,
and the matrix indexing actually stores end-to-end pathways instead of
just RAG chunks — which is what J meant by "why aren't we using it
for everything."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
5e8d87bf34 |
cleanup: remove unused HashSet import from 96b46cd + tighten applier gates
Some checks failed
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "journal event verified live (total_events_created 0→1 after probe)."
96b46cd ("first auto-applied commit") added `use tracing;` and
`use std::collections::HashSet;` to queryd/service.rs under a commit
message claiming to add a destructive SQL filter. HashSet was unused —
cargo check passed (warnings aren't errors) but the workspace now
carries a permanent `unused_imports` warning. `use tracing;` is
redundant but not flagged by the compiler, leave it.
This is an honest postmortem of the rationale-diff divergence problem:
emitter claimed one thing, diffed another. The cargo-green gate alone
can't catch that.
Applier hardening in this commit addresses all three failure modes:
- new-warning gate: reject patches that keep build green but add
warnings (baseline → post-patch diff)
- rationale-diff token alignment heuristic: reject patches whose
rationale shares no vocabulary with the actual new_string
- dry-run workspace revert: COMMIT=0 was silently leaving files
modified between runs; now reverts after each cargo check
- prompt additions: forbid unused-symbol imports; require rationale
vocabulary to appear in the diff
Next-iter applier runs should produce cleaner commits or none at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
25ea3de836 |
observer: fix LLM Team escalation — route to /v1/chat qwen3-coder:480b instead of dead mode
Some checks failed
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "journal event verified live (total_events_created 0→1 after probe)."
Discovery 2026-04-24: /api/run?mode=code_review returns "Unknown mode"
(error response from llm_team_ui.py). The 2026-04-24 observer escalation
wiring pointed at a dead endpoint and was failing silently. My earlier
claim of "9 registered LLM Team modes" came from GET probes that all
returned 405 — I interpreted that as "POST-only endpoints exist" when
it just means "GET is not allowed for anything, and on POST only `extract`
is registered."
Rewire: observer's escalateFailureClusterToLLMTeam now hits
POST /v1/chat { provider: "ollama_cloud", model: "qwen3-coder:480b", ... }
which is the same coding-specialist rung 2 of the scrum ladder that
reliably produces substantive reviews. Probe shows 1240 chars of
substantive analysis in ~8.7s.
Also tightens scrum_applier:
* MODEL default: kimi-k2:1t → qwen3-coder:480b (coding specialist)
* Size gate: 20 lines → 6 lines (surgical patches only)
* Max patches per file: 3 → 2
* Prompt: explicit forbidden-actions list (no struct renames, no
function-signature changes, no new modules) and mechanical-only
whitelist
These changes produced the first auto-applied commit (96b46cd), which
landed a 2-line import addition that passed cargo check. Zero-to-one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
8b77d67c9c |
OpenRouter rescue ladder + tree-split reduce fix + observer→LLM Team + scrum_applier + first auto-applied patch
Some checks failed
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "journal event verified live (total_events_created 0→1 after probe)."
## Infrastructure (scrum loop hardening)
crates/gateway/src/v1/openrouter.rs — new OpenRouter provider
Direct HTTPS to openrouter.ai/api/v1/chat/completions with OpenAI-compatible shape.
Key resolution: OPENROUTER_API_KEY env → /home/profit/.env → /root/llm_team_config.json
(shares LLM Team UI's quota). Added after iter 5 hit repeated Ollama Cloud 502s on
kimi-k2:1t — different provider backbone as rescue rung. Unit tests pin the URL
stripping and OpenAI wire shape.
crates/gateway/src/v1/mod.rs + main.rs
Added `"openrouter" | "openrouter_free"` arm to /v1/chat dispatch.
V1State.openrouter_key loaded at startup via openrouter::resolve_openrouter_key()
mirroring the Ollama Cloud pattern. Startup log:
"v1: OpenRouter key loaded — /v1/chat provider=openrouter enabled"
tests/real-world/scrum_master_pipeline.ts
* 9-rung ladder — kimi-k2:1t → qwen3-coder:480b → deepseek-v3.1:671b →
mistral-large-3:675b → gpt-oss:120b → qwen3.5:397b → openrouter/gpt-oss-120b:free
→ openrouter/gemma-3-27b-it:free → local qwen3.5:latest.
Added qwen3-coder:480b as rung 2 after live probes confirmed it rescues
kimi-k2:1t 502s cleanly (0.9s latency, substantive reviews).
Dropped devstral-2 (displaced by qwen3-coder); dropped kimi-k2.6 (not available);
dropped minimax-m2.7 (returned 0 chars / 400 thinking tokens).
Local fallback promoted qwen3.5:latest per J's direction 2026-04-24.
* MAX_ATTEMPTS bumped 6 → 9 to accommodate the rescue tier.
* Tree-split scratchpad fixed — was concatenating shard markers directly
into the reviewer input, causing kimi-k2:1t to write titles like
"Forensic Audit Report – file.rs (shard 3)". Now uses internal §N§
markers during accumulation and runs a proper reduce step that
collapses per-shard digests into ONE coherent file-level synthesis
with markers stripped. Matches the Phase 21 aibridge::tree_split
map→reduce design. Fallback to stripped scratchpad if reducer returns thin.
tests/real-world/scrum_applier.ts — NEW (737 lines)
The auto-apply pipeline. Reads scrum_reviews.jsonl, filters rows where
gradient_tier ∈ {auto, dry_run} AND confidence_avg ≥ MIN_CONF (default 90),
asks the reviewer model for concrete old_string/new_string patch JSON,
applies via text replacement, runs cargo check after each file, commits
if green and reverts if red. Deny-list: /etc/, config/, ops/, auditor/,
docs/, data/, mcp-server/, ui/, sidecar/, scripts/. Hard caps: per-patch
confidence ≥ MIN_CONF, old_string must be exactly unique, max 20 lines per
patch. Never runs on main without explicit LH_APPLIER_BRANCH override.
Audit trail in data/_kb/auto_apply.jsonl.
Empirical behavior (dry-run over iter 4 reviews):
5 eligible files → 1 green commit-ready, 2 build-red reverts, 2 all-rejected
The build-green gate caught 2 bad patches before they'd have merged.
mcp-server/observer.ts — LLM Team code_review escalation
When a sig_hash accumulates ≥3 failures (ESCALATION_THRESHOLD), fire-and-forget
POST /api/run?mode=code_review at localhost:5000 with the failure cluster context.
Parses facts/entities/relationships/file_hints from the response. Writes to a
new data/_kb/observer_escalations.jsonl surface. Answers J's vision of the
observer triggering richer LLM Team calls when failures pile up.
Non-blocking: runs parallel to existing qwen2.5 analyzer, never replaces it.
Tracks escalated sig_hashes in a session-local Set to avoid re-hammering
LLM Team when a cluster persists across observer cycles.
crates/aibridge/src/context.rs
First auto-applied patch produced by scrum_applier.ts (dry-run path —
applier writes files in dry-run mode but doesn't commit; bug noted for
iter 6 fix). Adds #[deprecated] annotation to the inline estimate_tokens
helper pointing callers to the centralized shared::model_matrix::ModelMatrix
entry point (P21-002 — duplicate token-estimator surfaces). Cargo check
passes with the annotation (verified by applier's own build gate).
## Visual Control Plane (UI)
ui/server.ts — Bun.serve on :3950 with /data/* fan-out:
/data/services, /data/reviews, /data/metrics, /data/trust, /data/overrides,
/data/findings, /data/outcomes, /data/audit_facts, /data/file/:path,
/data/refactor_signals, /data/search?q=, /data/signal_classes,
/data/logs/:svc (journalctl tail per systemd unit), /data/scrum_log.
Bug fix: tryFetch always attempts JSON.parse before falling back to text
— observer's Bun.serve returns JSON without application/json content-type,
which was displaying stats as a raw string ("0 ops" on map) before.
ui/index.html + ui.css — dark neo-brutalist shell. 6 views:
MAP (D3 force-graph + overlays) / TRACE (per-file iter history) /
TRAJECTORY (signal-class cards + refactor-signals table + reverse-index
search box) / METRICS (every card has SOURCE + GOOD lines explaining
where the number comes from and what target trajectory means) /
KB (card grid with tooltips on every field) / CONSOLE (per-service
journalctl tabs).
ui/ui.js — polling client, D3 wiring, signal-class panel, refactor-signals
table, reverse-index search, per-service console tabs. Bug fix:
renderNodeContext had Object.entries() iterating string characters when
/health returned a plain string — now guards with typeof check so
"lakehouse ok" renders as one row instead of "0 l / 1 a / 2 k / ...".
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
21fd3b9c61 |
Scrum-driven fixes: P5-001 auth wired, P42-001 truth evaluator, P9-001 journal on ingest
Some checks failed
lakehouse/auditor 2 blocking issues: cloud: claim not backed — "| **P9-001** (partial) | `crates/ingestd/src/service.rs` | **3 → 6** ↑↑↑ | `journal.record_ing
Apply the highest-confidence findings from the Phase 0→42 forensic sweep
after four scrum-master iterations under the adversarial prompt. Each fix
is independently validated by a later scrum iteration scoring the same
file higher under the same bar.
Code changes
────────────
P5-001 — crates/gateway/src/auth.rs + main.rs
api_key_auth was marked #[allow(dead_code)] and never wrapped around
the router, so `[auth] enabled=true` logged a green message and
enforced nothing. Now wired via from_fn_with_state, with constant-time
header compare and /health exempted for LB probes.
P42-001 — crates/truth/src/lib.rs
TruthStore::check() ignored RuleCondition entirely — signature looked
like enforcement, body returned every action unconditionally. Added
evaluate(task_class, ctx) that actually walks FieldEquals / FieldEmpty /
FieldGreater / Always against a serde_json::Value via dot-path lookup.
check() kept for back-compat. Tests 14 → 24 (10 new exercising real
pass/fail semantics). serde_json moved to [dependencies].
P9-001 (partial) — crates/ingestd/src/service.rs
Added Optional<Journal> to IngestState + a journal.record_ingest() call
on /ingest/file success. Gateway wires it with `journal.clone()` before
the /journal nest consumes the original. First-ever internal mutation
journal event verified live (total_events_created 0→1 after probe).
Iter-4 scrum scored these files higher under same prompt:
ingestd/src/service.rs 3 → 6 (P9-001 visible)
truth/src/lib.rs 3 → 4 (P42-001 visible)
gateway/src/auth.rs 3 → 4 (P5-001 visible)
gateway/src/execution_loop 4 → 6 (indirect)
storaged/src/federation 3 → 4 (indirect)
Infrastructure additions
────────────────────────
* tests/real-world/scrum_master_pipeline.ts
- cloud-first ladder: kimi-k2:1t → deepseek-v3.1:671b → mistral-large-3:675b
→ gpt-oss:120b → devstral-2:123b → qwen3.5:397b (deep final thinker)
- LH_SCRUM_FORENSIC env: injects SCRUM_FORENSIC_PROMPT.md as adversarial preamble
- LH_SCRUM_PROPOSAL env: per-iter fix-wave doc override
- Confidence extraction (markdown + JSON), schema v4 KB rows with:
verdict, critical_failures_count, verified_components_count,
missing_components_count, output_format, gradient_tier
- Model trust profile written per file-accept to data/_kb/model_trust.jsonl
- Fire-and-forget POST to observer /event so by_source.scrum appears in /stats
* mcp-server/observer.ts — unchanged in shape, confirmed receiving scrum events
* ui/ — new Visual Control Plane on :3950
- Bun.serve with /data/{services,reviews,metrics,trust,overrides,findings,file,refactor_signals,search,logs/:svc,scrum_log}
- Views: MAP (D3 graph, 5 overlays) / TRACE (per-file iter timeline) /
TRAJECTORY (refactor signals + reverse index search) / METRICS (explainers
with SOURCE + GOOD lines) / KB (card grid with tooltips) / CONSOLE (per-service
journalctl tail, tabs for gateway/sidecar/observer/mcp/ctx7/auditor/langfuse)
- tryFetch always attempts JSON.parse (fix for observer returning JSON without content-type)
- renderNodeContext primitive-vs-object guard (fix for gateway /health string)
* docs/SCRUM_FIX_WAVE.md — iter-specific scope directing the scrum
* docs/SCRUM_FORENSIC_PROMPT.md — adversarial audit prompt (verdict/critical/verified schema)
* docs/SCRUM_LOOP_NOTES.md — iteration observations + fix-next-loop queue
* docs/SYSTEM_EVOLUTION_LAYERS.md — Layers 1-10 roadmap (trust profiling, execution DNA, drift sentinel, etc)
Measurements across iterations
──────────────────────────────
iter 1 (soft prompt, gpt-oss:120b): mean score 5.00/10
iter 3 (forensic, kimi-k2:1t): mean score 3.56/10 (−1.44 — bar raised)
iter 4 (same bar, post fixes): mean score 4.00/10 (+0.44 — fixes landed)
Score movement iter3→iter4: ↑5 ↓1 =12
21/21 first-attempt accept by kimi-k2:1t in iter 4
20/21 emitted forensic JSON (richer signal than markdown)
16 verified_components captured (proof-of-life, new metric)
Permission Gradient distribution: 0 auto · 16 dry_run · 4 sim · 1 block
Observer loop: by_source {scrum: 21, langfuse: 1985, phase24_audit: 1}
v1/usage: 224 requests, 477K tokens, all tracked
Signal classes per file (iter 3 → iter 4):
CONVERGING: 1 (ingestd/service.rs — fix clearly landed)
LOOPING: 4 (catalogd/registry, main, queryd/service, vectord/index_registry)
ORBITING: 1 (truth — novel findings surfacing as surface ones fix)
PLATEAU: 9 (scores flat with high confidence — diminishing returns)
MIXED: 6
Loop thesis status
──────────────────
A file's score rises only when the scrum confirms a real fix landed.
No false positives yet across 3 iterations. Fixes applied to 3 files all
raised their independent scores under the same adversarial prompt. Loop
is measurable, not hand-wavy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
e2ccddd8d2 | Test updates: scenarios manifest + nine_consecutive_audits | ||
| 7c1745611a |
Audit pipeline PR #9: determinism + fact extraction + verifier gate + KB stats + context injection (PR #9)
Bundles PR #9's work for the audit pipeline: - N=3 consensus on cloud inference (gpt-oss:120b parallel) with qwen3-coder:480b tie-breaker - audit_discrepancies.jsonl logs N-run disagreements - scrum_master reviews route through llm_team fact extraction; source="scrum_review" - Verifier-gated persistence: drops INCORRECT, keeps UNVERIFIABLE/UNCHECKED; schema_version:2 - scrum_master_reviewed flag on accepted reviews - auditor/kb_stats.ts: on-demand observability script - claim_parser history/proof pattern class (verified-on-PR, was-flipping, the-proven-X) - claim_parser quoted-string guard (mirrors static.ts fix) - fact_extractor project context injection via docs/AUDITOR_CONTEXT.md - Fixed verifier-verdict parser to handle multiple gemma2 output formats Empirical: 3-run determinism test on unchanged PR #9 SHA showed 7/7 warn findings stable; block count oscillation eliminated; llm_team quality scores 8-9 on context-injected extract runs. See PR #9 for full run-by-run commit history. |
|||
| 156dae6732 |
Auditor self-test branch: real-world pipelines + cohesion Phase C + KB index (PR #8)
Bundles 12 commits validating the auditor + scrum_master architecture end-to-end: - enrich_prd_pipeline / hard_task_escalation / scrum_master_pipeline stress tests - Tree-split + scrum_reviews.jsonl + kb_query surfacing - Verdict → audit_lessons feedback loop (closed) - kb_index aggregator with confidence-based severity policy - 9-run + 5-run empirical tests proved the predictive-compounding property - Level 1 correction: temp=0 cloud inference for deterministic per-claim verdicts - audit_one.ts dry-run CLI - Fixes: static quoted-string guard, empirical-claim classification, symbol-resolver gate, repo-file size cap See PR #8 for run-by-run commit history. |
