6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
profit
|
9d12a814e3 |
auditor: kb_index aggregator + nine-consecutive empirical test
Some checks failed
lakehouse/auditor 1 blocking issue: cloud: claim not backed — "the proven escalation ladder with learning context, collects"
Phase 1 — definition-layer over append-only JSONL scratchpads.
auditor/kb_index.ts is the single shared aggregator:
aggregate<T>(jsonlPath, { keyFn, scopeFn, checkFn, tailLimit })
→ Map<signature, {count, distinct_scopes, confidence,
first_seen, last_seen, representative_summary, ...}>
ratingSeverity(agg) — confidence × count severity policy shared
across all KB readers. Kills the "same unfixed PR inflates its
own recurrence score" failure mode by design: confidence =
distinct_scopes/count, so same-scope noise stays below the 0.3
escalation threshold no matter how many times it repeats.
checkAuditLessons now routes through aggregate + ratingSeverity.
Net effect: the recurrence detector's bespoke Map/Set bookkeeping is
gone; same behavior, shared discipline, reusable by scrum/observer.
Also: symbolsExistInRepo now skips files >500KB so the audit can't
get stuck slurping a fixture.
Phase 2 — nine-consecutive audit runner.
tests/real-world/nine_consecutive_audits.ts pushes 9 empty commits,
waits for each verdict, captures the audit_lessons aggregate state
after each run, reports:
- sig_count trajectory (should stabilize, not grow linearly)
- max_count trajectory (same-signature repeat rate)
- max_confidence trajectory (must stay LOW on same-PR noise)
- verdict_stable across runs (must NOT oscillate)
This is the empirical proof that the KB compounds favorably:
noise doesn't escalate itself, and signal stays distinguishable.
Unit-tested both failure modes: same-PR × 9 repeats = conf=0.11
(info); cross-PR × 5 distinct = conf=1.00 (block). The rating
function correctly discriminates.
|
||
|
root
|
89d188074b |
scrum_master: tree-split + scrum_reviews.jsonl writer + truncation warning
Extends the scrum-master pipeline to handle input overflow on large source files (>6KB). Previously, the review prompt truncated the file to first-chunk, which caused false-positive "field is missing" findings whenever the actual field was past the cutoff. Now each file >FILE_TREE_SPLIT_THRESHOLD (6000) is sharded at FILE_SHARD_SIZE (3500), each shard summarized via gpt-oss:120b cloud, and the distillations merged into a scratchpad. The review then runs against the scratchpad with an explicit truncation-awareness clause in the prompt: "DO NOT claim any field, function, or feature is 'missing' based on its absence from this distillation." Also writes each accepted review as a JSONL row to data/_kb/scrum_reviews.jsonl (file, reviewed_at, accepted_model, accepted_on_attempt, attempts_made, tree_split_fired, preview). This is the source the auditor's kb_query reads to surface per-file scrum reviews on PRs that touch those files (cohesion plan Phase C). Verified: scrum review of 92KB playbook_memory.rs → 27 shards via cloud → distilled scratchpad → qwen3.5 local 7B accepted on attempt 1 (5931 chars). Tree-split fires, jsonl row appended, output file contains structured suggestions. |
||
|
profit
|
a7aba31935 |
tests/real-world: scrum-master pipeline — composes everything we built
The orchestrator J described: pulls git repo source + PRD +
suggested-changes doc, chunks them, hands each code piece through
the proven escalation ladder with learning context, collects
per-file suggestions in a consolidated handoff report.
Composes ONLY already-shipped primitives — no new core code:
- chunker with 800-char / 120-overlap windows
- sidecar /embed for real nomic-embed-text embeddings
- in-memory cosine retrieval for top-5 PRD + top-5 proposal
chunks per target file
- escalation ladder (qwen3.5 → qwen3 → gpt-oss:20b → gpt-oss:120b
→ devstral-2:123b → mistral-large-3:675b)
- per-attempt learning-context injection (prior failures as
"do not repeat" block)
- acceptance rubric (length ≥ 200 chars + structured form)
Live-run (tests/real-world/runs/scrum_moatqkee/):
targets: 3 files
- crates/vectord/src/playbook_memory.rs (920 lines)
- crates/vectord/src/doc_drift.rs (163 lines)
- auditor/audit.ts (170 lines)
resolved: 3/3 on attempt 1 by qwen3.5:latest local 7B
total duration: 111.7s
output: scrum_report.md + per-file JSON
Sample from scrum_report.md (playbook_memory.rs review):
- Alignment score: 9/10 vs PRD Phase 19
- 4 concrete change suggestions naming specific lines + PLAN/PRD
chunk offsets
- 3 gap analyses with PRD-reference citations
Honest findings from this run:
1. Local 7B handled review-style tasks first-try. The escalation
ladder infrastructure is live but didn't fire — review is an
easier task shape than strict code-generation (see hard_task
test which needed devstral-2 specialist).
2. 6KB file-truncation caused one false positive: model claimed
playbook_memory.rs lacks a `doc_refs` field, but that field
exists past the 6KB cutoff. Trade-off between context-size
and review-depth needs tuning per file.
3. Chunk-offset citations are real: model output includes
`[PRD @27880]` and `[PLAN @16320]` which map to the actual
byte offsets of retrieved context chunks. Auditor pattern could
adopt this for traceable claims.
This is the scrum-master-handoff shape J asked for:
repo + PRD + proposal → chunk → retrieve → escalate → consolidate
→ human-reviewable markdown report
Not shipping: per-PR diff analysis, open-PR integration, Gitea
posting of suggestions. Those compose the same primitives
differently — this proves the core pattern.
Env override: LH_SCRUM_FILES=path1,path2,... to target a different
file set. Default 3 files keeps runtime ~2min.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
profit
|
540c493ff1 |
tests/real-world: hard-task escalation — prove the ladder solves tasks local can't
J asked (2026-04-22): construct a task the local model provably can't
complete, then watch the escalation + retry + cloud pipeline actually
solve it.
The task: generate a Rust async function with 15 specific
structural rules (exact signature, bounded concurrency, exponential
backoff 250/500/1000ms, NO .unwrap(), rustdoc comments, etc.).
Small enough to fit in one response but strict enough that one
rule violation = not accepted. Fits Rust + async + concurrency +
error-handling — across the hardest dimensions for 7B models.
Escalation ladder (corrected per J — kimi-k2.x requires Ollama
Cloud Pro subscription which J's key lacks; mistral-large-3:675b
is the biggest provisioned model):
1. qwen3.5:latest (local 7B)
2. qwen3:latest (local 7B)
3. gpt-oss:20b (local 20B)
4. gpt-oss:120b (cloud 120B)
5. devstral-2:123b (cloud 123B coding specialist)
6. mistral-large-3:675b (cloud 675B — biggest available)
Each attempt gets PRIOR failures' rubric violations injected as
learning context. Loop caps at MAX_ATTEMPTS=6.
Live run (runs/hard_task_moapd3g3/):
attempt 1: qwen3.5:latest 11/15 — missed concurrency + some constraints
attempt 2: qwen3:latest 11/15 — different misses after learning
attempt 3: gpt-oss:20b 0/1 — empty response (local model dead-end)
attempt 4: gpt-oss:120b 0/1 — empty (heavy learning context may confuse)
attempt 5: devstral-2:123b 15/15 ✅ ACCEPTED after 10.4s
attempt 6: (not reached)
Total: 5 attempts, 145.6s, coding-specialist succeeded.
Honest findings from the run:
- Pipeline works: escalated through 4 distinct model tiers, injected
learning, bounded at 6, graceful failure surfaces.
- Learning injection doesn't always help general-purpose models —
gpt-oss:120b returned empty when given heavy prior-failure context
(attempt 4). The coding specialist (devstral) worked better because
the task is domain-aligned.
- Local 7B came within 4 rules of success first-try (11/15) — not
bad for the scale, but specific constraints like "EXACT signature"
and "bounded concurrency at 4" are where small models slip.
- Kimi K2.5/K2.6 both require a paid subscription on our current
Ollama Cloud key — verified via direct ollama.com curl. Swap
to kimi once subscription lands.
Also includes a rubric bug-fix caught in the run: the regex for
"reaches 500/1000ms backoff" originally required literal constants,
but devstral-2:123b wrote idiomatic `retry_delay *= 2;` which
doubles 250 → 500 → 1000 correctly. Broadened rubric to recognize
`*= 2`, bit-shift, `.pow()`, and literal forms. Without this the
ladder would have false-failed on semantically-correct code.
Files:
tests/real-world/hard_task_escalation.ts (270 LOC)
tests/real-world/runs/hard_task_moapd3g3/
attempt_{1..5}.txt — raw model outputs (last successful)
attempt_{1..5}.json — per-attempt rubric verdict + error
summary.json — ladder summary
What this PROVES that no prior test did:
- Task-level retry ESCALATES across distinct model capabilities
(not just same model retried)
- Bigger and more-specialized models ACTUALLY solve what smaller
ones can't — the ladder works by design, not by luck
- The subscription boundary (Kimi K2.x) is a real operational
constraint, not a code issue
- Rubric engineering is its own discipline — a strict-but-wrong
validator can reject correct code; shipping the test harness
required tuning against actual model outputs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
profit
|
6d6a306d4e |
tests/real-world: add task-level 6-retry loop (per J 2026-04-22)
Two distinct retry loops now both cap at 6 and serve different
purposes:
1. Per-cloud-call continuation (Phase 21 primitive) — when a single
cloud call returns empty or truncated, stitches up to 6
continuation calls. Handles output-overflow.
2. Per-TASK retry (this commit) — when the whole task errors
(500/404, thin answer, etc.), retries the full task up to 6
times. Each retry gets PRIOR ATTEMPTS' failures injected into
the prompt as learning context, so attempt N+1 is informed by
what N failed at. Handles error-recovery with compounding
context.
Both loops fired on iter 3 of the stress run, proving them
independent and composable:
FORCING TASK-RETRY LOOP — iter 3 will cycle through 5 invalid
models + 1 valid
attempt 1/6: model=deliberately-invalid-model-attempt-1
/v1/chat 502: ollama.com 404: model not found
attempt 2/6: [with prior-failure context]
... (5 failures total, each with the full chain of prior errors)
attempt 6/6: model=gpt-oss:20b [with prior-failure context]
continuation retry 1..6 (empty responses)
SUCCEEDED after 5 prior failures (441 chars)
What J was asking to prove:
"I expect it to retry the process six times to build on the
knowledge database... when an error is legitimately triggered
that it will go through six times... without getting caught in
a loop"
Proof:
- 6/6 attempts fired on the FORCED iteration
- Each retry embedded the preceding attempts' errors as "do not
repeat" context
- Hard cap at MAX_TASK_RETRIES (6) prevents infinite loops
- Last-ditch local fallback exists if all 6 still fail
- Other iterations succeed on attempt 1 — the loop ONLY fires
when errors are legitimately triggered
Stress run totals (runs/moan4h71/):
6/6 iterations complete, 58 cloud calls, 306s end-to-end
tree-splits: 6/6 continuations: 10 rescues: 2
iter 3: 8197+2800 tok, 6 task attempts, 6 continuation retries
local stored summary + per-iter JSON for inspection
What this proves that prior stress runs did NOT:
- Error-recovery at task granularity is live, not aspirational
- Compounding failure context flows between retries as text
- Loop bound is enforced; runaway cases aren't possible
- Two retry mechanisms compose without deadlock (continuation
inside task-retry inside tree-split)
Follow-ups worth doing (separate PRs):
- Persist retry-history to observer :3800 so cross-run learning
sees the failure patterns
- Route retries through /vectors/hybrid to surface similar prior
errors from the real KB (currently only in-memory across one
iteration)
- Fix citation regex in summary — iter 6 received 5 prior IDs
but counter shows 0 (regex needs to tolerate hyphens in IDs)
|
||
|
profit
|
4458c94f45 |
tests/real-world: enrich_prd_pipeline — architecture stress test
Real end-to-end test of the Lakehouse pipeline at scale. Runs the
PRD (63 KB, 901 lines → 93 chunks) through 6 iterations with cloud
inference, intentional failure injection, and tight context budget
to force every Phase 21 primitive to fire.
What the test exercises:
- Sidecar /embed for 93 chunks (nomic-embed-text)
- In-memory cosine retrieval for top-K per iteration
- Tree-split (shard → summarize → scratchpad → merge) when context
chunks exceed the 4000-char budget
- Scratchpad truncation to keep compounding context bounded
- Cloud inference via /v1/chat provider=ollama_cloud (gpt-oss:120b)
- Injected primary-cloud failure on iter 3 (invalid model name) +
rescue with gpt-oss:20b — proves catch-and-retry isn't dead code
- Playbook seeding per iteration (real HTTP against gateway)
- Prior-iteration answer injection for compounding (not just IDs —
the first version passed IDs only and the model ignored them)
Live run results (tests/real-world/runs/moamj810/):
6/6 iterations complete, 42 cloud calls total, 245s end-to-end
tree-splits: 6/6 (every iter overflowed 4K budget)
continuations: 0 (no responses hit max_tokens)
rescues: 1 (iter 3 injected failure → gpt-oss:20b → valid answer)
iter 6 answer explicitly cites [pb:pb-seed-82e1] — compounding real
scratchpad truncation fired on iter 6 as designed
What this PROVES:
- Tree-split primitives work under real context pressure, not just
in unit tests. The 4000-char budget forced every iteration to
shard 12 chunks → 6 shards → scratchpad → final answer.
- Rescue on primary failure is wired and produces answers from a
weaker model rather than erroring out.
- Compounding context injection works: iter 6's prompt had the 5
prior answers in its citation block, and the cloud model
acknowledged at least one via [pb:...] notation.
- The existence claims in Phase 21 (continuation + tree-split) are
backed by executable evidence, not just unit tests.
What this DOESN'T prove (deliberate — scoped for follow-up):
- Continuation retries (no iter hit max_tokens in this run; would
need a harder prompt or lower max_tokens to force)
- Real integration with /vectors/hybrid endpoint (test does in-memory
cosine instead, bypassing gateway vector surface)
- Observer consumption of these runs (nothing posted to :3800 during
the test — adding that is Phase A integration, handled separately)
Files:
tests/real-world/enrich_prd_pipeline.ts (333 LOC)
tests/real-world/runs/moamj810/{iter_1..6.json, summary.json}
— artifacts from the stress run, committed for inspection
Follow-ups worth doing:
1. Lower max_tokens / harder prompt to force continuation path
2. Route retrieval through /vectors/hybrid for real Phase 19 boost
3. POST per-iteration summary to observer :3800 so runs accumulate
like scenario runs do
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|