J asked (2026-04-22): construct a task the local model provably can't
complete, then watch the escalation + retry + cloud pipeline actually
solve it.
The task: generate a Rust async function with 15 specific
structural rules (exact signature, bounded concurrency, exponential
backoff 250/500/1000ms, NO .unwrap(), rustdoc comments, etc.).
Small enough to fit in one response but strict enough that one
rule violation = not accepted. Fits Rust + async + concurrency +
error-handling — across the hardest dimensions for 7B models.
Escalation ladder (corrected per J — kimi-k2.x requires Ollama
Cloud Pro subscription which J's key lacks; mistral-large-3:675b
is the biggest provisioned model):
1. qwen3.5:latest (local 7B)
2. qwen3:latest (local 7B)
3. gpt-oss:20b (local 20B)
4. gpt-oss:120b (cloud 120B)
5. devstral-2:123b (cloud 123B coding specialist)
6. mistral-large-3:675b (cloud 675B — biggest available)
Each attempt gets PRIOR failures' rubric violations injected as
learning context. Loop caps at MAX_ATTEMPTS=6.
Live run (runs/hard_task_moapd3g3/):
attempt 1: qwen3.5:latest 11/15 — missed concurrency + some constraints
attempt 2: qwen3:latest 11/15 — different misses after learning
attempt 3: gpt-oss:20b 0/1 — empty response (local model dead-end)
attempt 4: gpt-oss:120b 0/1 — empty (heavy learning context may confuse)
attempt 5: devstral-2:123b 15/15 ✅ ACCEPTED after 10.4s
attempt 6: (not reached)
Total: 5 attempts, 145.6s, coding-specialist succeeded.
Honest findings from the run:
- Pipeline works: escalated through 4 distinct model tiers, injected
learning, bounded at 6, graceful failure surfaces.
- Learning injection doesn't always help general-purpose models —
gpt-oss:120b returned empty when given heavy prior-failure context
(attempt 4). The coding specialist (devstral) worked better because
the task is domain-aligned.
- Local 7B came within 4 rules of success first-try (11/15) — not
bad for the scale, but specific constraints like "EXACT signature"
and "bounded concurrency at 4" are where small models slip.
- Kimi K2.5/K2.6 both require a paid subscription on our current
Ollama Cloud key — verified via direct ollama.com curl. Swap
to kimi once subscription lands.
Also includes a rubric bug-fix caught in the run: the regex for
"reaches 500/1000ms backoff" originally required literal constants,
but devstral-2:123b wrote idiomatic `retry_delay *= 2;` which
doubles 250 → 500 → 1000 correctly. Broadened rubric to recognize
`*= 2`, bit-shift, `.pow()`, and literal forms. Without this the
ladder would have false-failed on semantically-correct code.
Files:
tests/real-world/hard_task_escalation.ts (270 LOC)
tests/real-world/runs/hard_task_moapd3g3/
attempt_{1..5}.txt — raw model outputs (last successful)
attempt_{1..5}.json — per-attempt rubric verdict + error
summary.json — ladder summary
What this PROVES that no prior test did:
- Task-level retry ESCALATES across distinct model capabilities
(not just same model retried)
- Bigger and more-specialized models ACTUALLY solve what smaller
ones can't — the ladder works by design, not by luck
- The subscription boundary (Kimi K2.x) is a real operational
constraint, not a code issue
- Rubric engineering is its own discipline — a strict-but-wrong
validator can reject correct code; shipping the test harness
required tuning against actual model outputs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two distinct retry loops now both cap at 6 and serve different
purposes:
1. Per-cloud-call continuation (Phase 21 primitive) — when a single
cloud call returns empty or truncated, stitches up to 6
continuation calls. Handles output-overflow.
2. Per-TASK retry (this commit) — when the whole task errors
(500/404, thin answer, etc.), retries the full task up to 6
times. Each retry gets PRIOR ATTEMPTS' failures injected into
the prompt as learning context, so attempt N+1 is informed by
what N failed at. Handles error-recovery with compounding
context.
Both loops fired on iter 3 of the stress run, proving them
independent and composable:
FORCING TASK-RETRY LOOP — iter 3 will cycle through 5 invalid
models + 1 valid
attempt 1/6: model=deliberately-invalid-model-attempt-1
/v1/chat 502: ollama.com 404: model not found
attempt 2/6: [with prior-failure context]
... (5 failures total, each with the full chain of prior errors)
attempt 6/6: model=gpt-oss:20b [with prior-failure context]
continuation retry 1..6 (empty responses)
SUCCEEDED after 5 prior failures (441 chars)
What J was asking to prove:
"I expect it to retry the process six times to build on the
knowledge database... when an error is legitimately triggered
that it will go through six times... without getting caught in
a loop"
Proof:
- 6/6 attempts fired on the FORCED iteration
- Each retry embedded the preceding attempts' errors as "do not
repeat" context
- Hard cap at MAX_TASK_RETRIES (6) prevents infinite loops
- Last-ditch local fallback exists if all 6 still fail
- Other iterations succeed on attempt 1 — the loop ONLY fires
when errors are legitimately triggered
Stress run totals (runs/moan4h71/):
6/6 iterations complete, 58 cloud calls, 306s end-to-end
tree-splits: 6/6 continuations: 10 rescues: 2
iter 3: 8197+2800 tok, 6 task attempts, 6 continuation retries
local stored summary + per-iter JSON for inspection
What this proves that prior stress runs did NOT:
- Error-recovery at task granularity is live, not aspirational
- Compounding failure context flows between retries as text
- Loop bound is enforced; runaway cases aren't possible
- Two retry mechanisms compose without deadlock (continuation
inside task-retry inside tree-split)
Follow-ups worth doing (separate PRs):
- Persist retry-history to observer :3800 so cross-run learning
sees the failure patterns
- Route retries through /vectors/hybrid to surface similar prior
errors from the real KB (currently only in-memory across one
iteration)
- Fix citation regex in summary — iter 6 received 5 prior IDs
but counter shows 0 (regex needs to tolerate hyphens in IDs)
Real end-to-end test of the Lakehouse pipeline at scale. Runs the
PRD (63 KB, 901 lines → 93 chunks) through 6 iterations with cloud
inference, intentional failure injection, and tight context budget
to force every Phase 21 primitive to fire.
What the test exercises:
- Sidecar /embed for 93 chunks (nomic-embed-text)
- In-memory cosine retrieval for top-K per iteration
- Tree-split (shard → summarize → scratchpad → merge) when context
chunks exceed the 4000-char budget
- Scratchpad truncation to keep compounding context bounded
- Cloud inference via /v1/chat provider=ollama_cloud (gpt-oss:120b)
- Injected primary-cloud failure on iter 3 (invalid model name) +
rescue with gpt-oss:20b — proves catch-and-retry isn't dead code
- Playbook seeding per iteration (real HTTP against gateway)
- Prior-iteration answer injection for compounding (not just IDs —
the first version passed IDs only and the model ignored them)
Live run results (tests/real-world/runs/moamj810/):
6/6 iterations complete, 42 cloud calls total, 245s end-to-end
tree-splits: 6/6 (every iter overflowed 4K budget)
continuations: 0 (no responses hit max_tokens)
rescues: 1 (iter 3 injected failure → gpt-oss:20b → valid answer)
iter 6 answer explicitly cites [pb:pb-seed-82e1] — compounding real
scratchpad truncation fired on iter 6 as designed
What this PROVES:
- Tree-split primitives work under real context pressure, not just
in unit tests. The 4000-char budget forced every iteration to
shard 12 chunks → 6 shards → scratchpad → final answer.
- Rescue on primary failure is wired and produces answers from a
weaker model rather than erroring out.
- Compounding context injection works: iter 6's prompt had the 5
prior answers in its citation block, and the cloud model
acknowledged at least one via [pb:...] notation.
- The existence claims in Phase 21 (continuation + tree-split) are
backed by executable evidence, not just unit tests.
What this DOESN'T prove (deliberate — scoped for follow-up):
- Continuation retries (no iter hit max_tokens in this run; would
need a harder prompt or lower max_tokens to force)
- Real integration with /vectors/hybrid endpoint (test does in-memory
cosine instead, bypassing gateway vector surface)
- Observer consumption of these runs (nothing posted to :3800 during
the test — adding that is Phase A integration, handled separately)
Files:
tests/real-world/enrich_prd_pipeline.ts (333 LOC)
tests/real-world/runs/moamj810/{iter_1..6.json, summary.json}
— artifacts from the stress run, committed for inspection
Follow-ups worth doing:
1. Lower max_tokens / harder prompt to force continuation path
2. Route retrieval through /vectors/hybrid for real Phase 19 boost
3. POST per-iteration summary to observer :3800 so runs accumulate
like scenario runs do
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
profit