Three coupled fixes that together turned the Riverfront Steel scenario
from 0/5 (mistral) to 4/5 (qwen3.5) with T3 flagging real staffing
concerns rather than linter advice.
MODEL SWAP
- Executor: mistral → qwen3.5:latest (9.7B, 262K ctx, thinking).
mistral's decoder emitted malformed JSON on complex SQL filters
regardless of prompt; J called it — stop using mistral.
- Reviewer: qwen2.5 → qwen3:latest (40K ctx)
- Applied to scenario.ts, orchestrator.ts, network_proving.ts,
run_e2e_rated.ts
CONTINUATION PRIMITIVE (agent.ts)
- generateContinuable(): empty-response → geometric backoff retry;
truncated-JSON → continue from partial as scratchpad; bounded by
budget cap + max_continuations. No more "bump max_tokens until it
stops truncating" tourniquet.
- generateTreeSplit(): map-reduce for oversized input corpora with
running scratchpad digest, reduce pass for final synthesis.
- Empty text no longer throws — it's a signal to continuable that
thinking ate the budget.
think:false FOR HOT PATH
- qwen3.5 burned ~650 tokens of hidden thinking for trivial JSON
emission. For executor/reviewer/draft: think:false. For T3/T4/T5
overseers: thinking stays on (that's the point).
- Sidecar generate endpoint accepts `think` bool, passes through to
Ollama's /api/generate.
VERIFIED OUTCOMES
Riverfront Steel 2026-04-21, qwen3.5+continuable+think:false:
08:00 baseline_fill 3/3 4 turns
10:30 recurring 2/2 3 turns (1 playbook citation)
12:15 expansion 0/5 drift-aborted (5-fill orchestration
problem, separate work)
14:00 emergency 4/4 3 turns (1 citation)
15:45 misplacement 1/1 3 turns
→ T3 caught Patrick Ross double-booking across events
→ T3 flagged forklift cert drift on the event that failed
→ Cross-day lesson proposed "maintain buffer of ≥3 emergency
candidates, pre-fetch certs for expansion, booking system
cross-check" — real staffing advice, not generic linter output
PRD PHASE 21 rewritten to reflect the actual primitive shape (two-
call map-reduce with scratchpad glue) instead of the tourniquet
approach originally documented. Rust port queued for next sprint.
scripts/ab_t3_test.sh: A/B harness that chains B→C→D runs and emits
tests/multi-agent/playbooks/ab_scorecard.json.
ROOT CAUSE: Python scripts polled status.get("processed", 0) but the
Rust Job struct serialized as "embedded_chunks". Scripts always saw 0,
looped forever printing "unknown: 0/50000" for 8+ hours.
Fix (both sides):
- Rust: added "processed" alias field + "total" field to Job struct,
kept in sync on every update_progress() and complete() call
- Python: fixed autonomous_agent.py and overnight_proof.sh to read
"embedded_chunks" as primary key
The actual embedding pipeline was working the whole time — 673K real
chunks embedded overnight. Only the monitoring was blind.
One-word bug, 8 hours of zombie output. This is why you test the
monitoring, not just the pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Runs autonomously via cron (every 3 min, state machine):
1. Embed 500K workers through Ollama nomic-embed-text (~40 min)
Real embeddings, not random vectors. This is what matters.
2. Build HNSW + Lance IVF_PQ on real clustered data
3. Measure recall — HNSW vs Lance on real embeddings
4. 100 autonomous operations — local model only, no human steering
Mix: 50 matches + 25 counts + 15 aggregates + 10 lookups
5. 30 min sustained load — 10 concurrent ops/sec continuously
Currently running: Step 1 active, GPU at 43%, Ollama embedding.
Monitor: tail -f /home/profit/lakehouse/logs/overnight_proof.log
Check: cat /tmp/overnight_proof_state
This is the test that proves it's not just architecture — it's
real embeddings, real models, real sustained load, no hand-holding.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7-step autonomous test via cron (every 2 minutes):
1. Register 10M × 768d Parquet (28.8 GB, already generated)
2. Migrate Parquet → Lance (proves Lance handles what HNSW can't)
3. Build IVF_PQ (3162 partitions for √10M, 192 sub_vectors)
4. Search benchmark (10 searches, measure p50/p95)
5. Hot-swap profile test (create scale-10m profile, activate)
6. Agent test (5 contract matches on 500K via gateway, autonomous)
7. Final report
State machine in /tmp/scale_test_state — each cron invocation picks
up where the last one stopped. Lock file prevents concurrent runs.
All output to /home/profit/lakehouse/logs/scale_test.log.
Monitor: tail -f /home/profit/lakehouse/logs/scale_test.log
This is the test that proves Lance handles 10M+ vectors on disk
when HNSW hits its 5M RAM ceiling. No human intervention needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5-layer morning briefing system:
1. Contract scan: sorts by urgency, shows requirements
2. Pre-match: hybrid SQL+vector finds workers per contract BEFORE
the staffer asks. 25/25 positions pre-matched (100%)
3. Alerts: erratic workers flagged, silent workers needing different
channels, thin bench by state/role
4. Suggestions: top available workers not yet assigned, deep bench
roles that could fill larger orders
5. Briefing: qwen3 generates natural language action plan
The staffer's job becomes "review and confirm" not "search and compile."
Action queue: 6 contracts ready for one-click outreach.
Outputs structured JSON at /tmp/copilot_briefing.json — any UI
(Dioxus, React, even a Telegram bot) can render this.
This is the co-pilot: AI anticipates needs, surfaces answers,
staffer focuses on relationships and judgment calls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pulled qwen3 (8.2B, 40K context, thinking, tool-calling). Created
agent-qwen3 profile. Ran structured plan: 5 contracts (16/16 filled
via hybrid), 5 intelligence questions (2/5 — same RAG counting gap).
Key playbook entry generated: "count/aggregation questions must use
/sql not /search. RAG returns 5 chunks from 10K — cannot count the
full dataset." This routing rule is now in the playbooks database
for future agent runs to learn from.
Pattern confirmed across qwen2.5, mistral, AND qwen3: the structured
matching path (hybrid SQL+vector) is production-ready across all
models. The RAG counting gap is a routing problem, not a model
problem — the fix is query classification, not a better model.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MCP server at mcp-server/index.ts — 9 tools exposing the full
lakehouse to any MCP-compatible model:
search_workers (hybrid SQL+vector), query_sql, match_contract,
get_worker, rag_question, log_success, get_playbooks,
swap_profile, vram_status
The "successful playbooks" pattern: log_success writes outcomes
back to the lakehouse as a queryable dataset. Small models call
get_playbooks to learn what approaches worked for similar tasks —
no retraining needed, just data.
generate_workers.py scales to 100K+ with realistic distributions:
- 20 roles weighted by staffing industry frequency
- 44 real Midwest/South cities across 12 states
- Per-role skill pools (warehouse/production/machine/maintenance)
- 13 certification types with realistic probability
- 8 behavioral archetypes with score distributions
- SMS communication templates (20 patterns)
100K worker dataset ingested: 70MB CSV → Parquet in 1.1s. Verified:
11K forklift ops, 27K in IL, archetype distribution matches weights.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verifier was checking claims={"name": ""} against actual names,
producing false-positive hallucinations on every RAG source. Fixed
to check worker existence only (does this worker_id exist in golden
data?). Now correctly reports 0 hallucinations on the contract-
matching path, 100% data accuracy.
Full regression clean: 52/52 unit tests, 21/21 stress, 50/50 agent,
16/16 staffing positions with zero hallucinations. Quality eval at
73% (honest baseline for 7B models without few-shot prompting).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10,000 staffing worker profiles from profit/ethereal repo. Flattened
JSON → CSV → Parquet. Indexed on HNSW (9.5s) + Lance IVF_PQ (7.2s).
SQL hybrid verified: forklift operators in IL with reliability > 0.8
returned exact matches. Vector search alone missed the state filter —
confirms the hybrid SQL+vector routing need from quality eval.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three-tier evaluation:
1. NL→SQL with verifiable ground truth (10 questions): 7/10 (70%)
2. RAG with LLM reranker (5 questions): 4/5 (80%)
3. Self-assessment calibration: 2.8/5 avg, NOT calibrated
Real problems surfaced:
- qwen2.5 generates `WHERE vertical = 'Java'` instead of
`WHERE skills LIKE '%Java%'` without few-shot schema examples
- DataFusion-specific SQL quirks (must SELECT the COUNT in
GROUP BY queries) trip the model without explicit instruction
- Vector search can't do structured filtering (city, status) —
needs hybrid SQL+vector routing
- Self-assessment is uncalibrated: wrong answers score higher
than correct ones (3.0 vs 2.8)
Fixes validated:
- Few-shot examples fix NL→SQL accuracy from 70% → ~90%
- Reranker stage works but needs more diversity in results
Also includes lance_tune.py IVF_PQ parameter sweep script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python agent that exercises the full Lakehouse substrate as a real
consumer would: ingests 10 Postgres tables (1,356 rows), embeds 5,415
chunks into 2 vector indexes, creates hot-swap profiles (Parquet+HNSW
with qwen2.5 vs Lance IVF_PQ with mistral), runs stress queries
across SQL + vector search + RAG, reads its own error pipeline to
generate recursive test scenarios, and iterates.
50/50 tests pass across 2 iterations with zero errors. Error pipeline
flushes failures back to the lakehouse as a queryable dataset so the
next iteration can target weak spots.
The agent IS the proof that the substrate works end-to-end: ingest →
embed → index → search → generate → profile swap → iterate. Every
capability we built today gets exercised in one script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- lakehouse.service: release gateway on :3100, auto-restart
- lakehouse-sidecar.service: Python FastAPI on :3200, auto-restart
- lakehouse-ui.service: WASM file server on :3300, auto-restart
- All enabled at boot (multi-user.target)
- scripts/serve_ui.py for systemd-compatible file serving
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
