140 Commits

Author	SHA1	Message	Date
root	ba928b1d64	aibridge: drop Python sidecar from hot path; AiClient → direct Ollama ... The "drop Python sidecar from Rust aibridge" item from the architecture_comparison decisions tracker. Universal-win cleanup — removes 1 process + 1 runtime + 1 hop from every embed/generate request, with no behavior change. ## What was on the hot path before gateway → AiClient → http://:3200 (FastAPI sidecar) ├── embed.py → http://:11434 (Ollama) ├── generate.py → http://:11434 ├── rerank.py → http://:11434 (loops generate) └── admin.py → http://:11434 (/api/ps + nvidia-smi) The sidecar's hot-path code (~120 LOC across embed.py / generate.py / rerank.py / admin.py) was pure pass-through: each route translated its request body to Ollama's wire format and returned Ollama's response in a sidecar envelope. Zero logic, one full HTTP hop of overhead. ## What's on the hot path now gateway → AiClient → http://:11434 (Ollama directly) Inline rewrites in crates/aibridge/src/client.rs: - embed_uncached: per-text loop to /api/embed; computes dimension from response[0].length (matches the sidecar's prior shape) - generate (direct path): translates GenerateRequest → /api/generate (model, prompt, stream:false, options:{temperature, num_predict}, system, think); maps response → GenerateResponse using Ollama's field names (response, prompt_eval_count, eval_count) - rerank: per-doc loop with the same score-prompt the sidecar used; parses leading number, clamps 0-10, sorts desc - unload_model: /api/generate with prompt:"", keep_alive:0 - preload_model: /api/generate with prompt:" ", keep_alive:"5m", num_predict:1 - vram_snapshot: GET /api/ps + std::process::Command nvidia-smi; same envelope shape as the sidecar's /admin/vram so callers keep parsing - health: GET /api/version, wrapped in a sidecar-shaped envelope ({status, ollama_url, ollama_version}) Public AiClient API is unchanged — Request/Response types untouched. Callers (gateway routes, vectord, etc.) require zero updates. ## Config changes - crates/shared/src/config.rs: default_sidecar_url() bumps to :11434. The TOML field stays `[sidecar].url` for migration compat (operators with existing configs don't need to rename anything). - lakehouse.toml + config/providers.toml: bumped to localhost:11434 with comments explaining the 2026-05-02 transition. ## What stays Python sidecar/sidecar/lab_ui.py (385 LOC) + pipeline_lab.py (503 LOC) are dev-mode Streamlit-shape UIs for prompt experimentation. Not on the runtime hot path; continue running for ad-hoc work. The embed/generate/rerank/admin routes inside sidecar can be retired, but operators who want to keep the sidecar process running for the lab UI face no breakage — those routes still call Ollama and work. ## Verification - cargo check --workspace: clean - cargo test -p aibridge --lib: 32/32 PASS - Live smoke against test gateway on :3199 with new config: /ai/embed → 768-dim vector for "forklift operator" ✓ /v1/chat → provider=ollama, model=qwen2.5:latest, content=OK ✓ - nvidia-smi parsing tested via std::process::Command path - Live `lakehouse.service` (port :3100) NOT yet restarted — deploy step is operator-driven (sudo systemctl restart lakehouse.service) ## Architecture comparison update (Captured separately in golangLAKEHOUSE/docs/ARCHITECTURE_COMPARISON.md decisions tracker.) The "drop Python sidecar" line moves from _open_ to DONE. The Rust process model now has 1 mega-binary instead of 1 mega-binary + 1 sidecar process — a small but real reduction in ops surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-02 04:59:47 -05:00
root	654797a429	gateway: pub extract_json + parity_extract_json bin (cross-runtime probe) ... Supports the 2026-05-02 cross-runtime parity probe at golangLAKEHOUSE/scripts/cutover/parity/extract_json_parity.sh which feeds identical model-output strings through both runtimes' extract_json and diffs results. ## Changes - crates/gateway/src/v1/iterate.rs: extract_json gains `pub` + a comment pointing at the Go counterpart and the parity probe path - crates/gateway/src/lib.rs: NEW thin lib facade re-exporting the modules so sub-binaries can reuse them. main.rs is unchanged (still uses local mod declarations) - crates/gateway/src/bin/parity_extract_json.rs: NEW ~30-LOC binary that reads stdin, calls extract_json, prints {matched, value} JSON ## Probe result (logged in golangLAKEHOUSE) 12/12 match across fenced blocks, nested objects, unicode, escaped quotes, top-level array, malformed JSON. Both runtimes' algorithms are genuinely equivalent. Substrate gate the probe enforces: `cargo test -p gateway extract_json` PASS before any parity comparison runs. So a future divergence in the live extract_json fires either as a Rust test failure (live behavior changed) or a probe diff (Go behavior changed) — never silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-02 04:44:11 -05:00
root	150cc3b681	aibridge: LRU embed cache - 236x RPS gain on warm workloads. Per architecture_comparison.md universal-win for Rust side. Cache key (model,text), default 4096 entries, in-process inside gateway. Load test: 128 RPS -> 30k+ RPS, p50 78ms -> 129us.	2026-05-01 04:45:20 -05:00
root	8de94eba08	cleanup: bump qwen2.5 → qwen3.5:latest in active defaults ... stronger local rung is now the small-model-pipeline tier-1 default across both Rust legacy + Go rewrite (cf. golangLAKEHOUSE phase 1). same JSON-clean property as qwen2.5, more capacity. ollama still serves both side-by-side; rollback is a 4-line revert if a workload regresses. active-default sites: - lakehouse.toml [ai] gen_model + rerank_model → qwen3.5:latest - mcp-server/observer.ts diagnose call (Phase 44 /v1/chat path) → qwen3.5:latest - mcp-server/index.ts model roster doc → qwen3.5:latest first - crates/vectord/src/rag.rs ContinuableOpts + RagResponse.model → qwen3.5:latest skipped: execution_loop/mod.rs comments describing historic qwen2.5 tool_call quirks — those are documentation of past behavior, not active defaults. data/_catalog/profiles/*.json are runtime-generated (gitignored), not in scope for tracked changes. cargo check -p vectord: clean. no behavioral change in the audit pipeline — same JSON-clean local model, same think=Some(false) posture, just stronger upstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-30 00:10:57 -05:00
root	d475fc7fff	infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths ... Ollama Pro plan went live today (39-model fleet on the same OLLAMA_CLOUD_KEY) and OpenCode Zen was already wired in the gateway but not consumed. Routing every gpt-oss call site to faster / stronger replacements: | Site | gpt-oss → replacement | Why | |---|---|---| | ollama_cloud default | gpt-oss:120b → deepseek-v3.2 | newest DeepSeek revision; live-probed `pong` | | openrouter default | openai/gpt-oss-120b:free → x-ai/grok-4.1-fast | already the scrum LADDER's PRIMARY | | modes.toml staffing_inference | openai/gpt-oss-120b:free → kimi-k2.6 | coding-specialized, on Ollama Pro | | modes.toml doc_drift_check | gpt-oss:120b → gemini-3-flash-preview | speed leader for factual checks | | scrum_master_pipeline tree-split MAP+REDUCE | gpt-oss:120b → gemini-3-flash-preview | latency-dominated path (5-20× per file) | | bot/propose.ts CLOUD_MODEL | gpt-oss:120b → deepseek-v3.2 | same Ollama key, faster | | mcp-server/observer.ts overseer label fallback | gpt-oss:120b → claude-opus-4-7 | matches new overseer model | | crates/gateway/src/execution_loop overseer escalation | ollama_cloud/gpt-oss:120b → opencode/claude-opus-4-7 | frontier reasoning matters here — fires only after local self-correct fails twice; Zen pay-per-token cost is bounded | Verification: - `cargo check -p gateway --tests` — clean - Live probes through localhost:3100/v1/chat: - `opencode/claude-opus-4-7` → "pong" - `gemini-3-flash-preview` (ollama_cloud) → "pong" - `kimi-k2.6` (ollama_cloud) → "pong" - `deepseek-v3.2` (ollama_cloud) → "Pong! 🏓" Notes: - kimi-k2:1t still upstream-broken (HTTP 500 on Ollama Pro probe today, matches yesterday's memory). Replacement table never picks it. - The Rust changes need a `systemctl restart lakehouse.service` to take effect on the running gateway. TS callers reload on next run. - aibridge/src/context.rs still has gpt-oss:{20b,120b} in its window- size lookup table; harmless and kept for callers that pass it explicitly as an override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-28 06:13:30 -05:00
root	6366487b45	ops: persist runtime fixes — iterate.rs unused state, catalog cleanup ... Two load-bearing runtime changes that were never committed: 1. crates/gateway/src/v1/iterate.rs — `state` → `_state` on the unused route-state parameter. Cleared the one cargo workspace warning. Fix was made earlier this session but the working-tree change never made it into a commit. 2. data/_catalog/manifests/564b00ae-cbf3-4efd-aa55-84cdb6d2b0b7.json — DELETED. This was the dead manifest for `client_workerskjkk`, a typo dataset whose parquet was deleted but whose catalog entry stayed registered. Every SQL query failed schema inference on the missing file before reaching its target table — that's the bug that made /system/summary report 0 workers and the demo show zero bench. Deleting the manifest keeps the fix on disk; committing the deletion keeps it in git so a fresh checkout doesn't regress. 3. data/_catalog/manifests/32ee74a0-59b4-4e5b-8edb-70c9347a4bf3.json — runtime catalog metadata update from the successful_playbooks_live write path. Ride-along change. Reports under reports/distillation/phase[68]-*.md are auto-regenerated by the audit cycle each run; skipping those.	2026-04-28 06:01:04 -05:00
root	6ed48c1a69	gateway+validator: /v1/health reports honest worker count for production ... Adds `fn len() -> usize` (default 0) to the WorkerLookup trait. The InMemoryWorkerLookup overrides with HashMap size; ParquetWorkerLookup constructs an InMemoryWorkerLookup so it inherits the count. /v1/health now reports `workers_count` (exact integer) alongside `workers_loaded` (derived bool: count > 0). The previous placeholder true was a known caveat in the prior commit's body — this closes it. Production switchover use case: J swaps workers_500k.parquet → real Chicago contractor data, restarts the gateway, and verifies the swap with one curl: curl http://localhost:3100/v1/health | jq .workers_count Expected: matches the row count of the new file. Mismatch (or 0) means the file is missing / unreadable / had a schema mismatch and the gateway fell back to the empty InMemoryWorkerLookup. Operator catches the drift before traffic reaches the validators. Verified live (current synthetic data): workers_count: 500000 (matches workers_500k.parquet row count) workers_loaded: true When the Chicago data lands, the same curl is the single source of truth that the new dataset is hot. Removes the restart-and-pray failure mode. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 08:07:18 -05:00
root	74ad77211f	gateway: /v1/health — production operational status endpoint ... Adds GET /v1/health that returns a JSON snapshot of subsystem state so operators (and load balancers, and the lakehouse-auditor service) can verify the gateway is fully booted before routing traffic. Phase 42-45 closures are now production-deployable; this endpoint is the canary that proves it. Returns 200 always — fields are observed-state, not pass/fail gates. Monitoring tools evaluate the booleans + counts against their own thresholds. Shape: { "status": "ok", "workers_loaded": bool, "providers_configured": { "ollama_cloud": bool, "openrouter": bool, "kimi": bool, "opencode": bool, "gemini": bool, "claude": bool, }, "langfuse_configured": bool, "usage_total_requests": N, "usage_by_provider": ["ollama_cloud", "openrouter", ...] } Verified live: curl http://localhost:3100/v1/health → 4 providers configured (kimi, ollama_cloud, opencode, openrouter) → 2 not configured (claude, gemini — keys not wired) → langfuse_configured: true → workers_loaded: true (500K-row workers_500k.parquet snapshot) Caveat: workers_loaded is a placeholder true — WorkerLookup trait doesn't have a len() method yet, so we can't honestly report row count from the runtime probe. The boot log line "loaded workers parquet snapshot rows=N" is the source of truth on count. Future follow-up: add `fn len(&self) -> usize` to WorkerLookup so /v1/health can report the exact figure. Pre-production checklist context: J flagged production switchover incoming — synthetic profiles will be replaced with real Chicago data soon. /v1/health gives the operator a single curl to verify the gateway sees the new data after the parquet swap (boot log + this endpoint). Hot-swap reload (POST /v1/admin/reload-workers) deferred to a follow-up — requires V1State.validate_workers to wrap in RwLock or ArcSwap so write traffic doesn't block the steady-state read path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 08:05:52 -05:00
root	6cafa7ec0e	vectord: Phase 45 closure — /doc_drift/scan + doc_drift_corrections.jsonl writes ... Phase 45 (doc-drift detection + context7 integration) was mostly already shipped in prior sessions: DocRef struct, doc_drift module, /doc_drift/check + /doc_drift/resolve endpoints, mcp-server's context7_bridge.ts, boost exclusion in compute_boost_for_filtered _with_role. The two missing pieces this commit lands: 1. POST /vectors/playbook_memory/doc_drift/scan — batch scan across ALL active playbooks. Iterates the snapshot, filters out retired + already-flagged + no-doc_refs, runs check_all_refs on the rest, flags drifted entries via PlaybookMemory::flag_doc_drift. 2. Per-detection write to data/_kb/doc_drift_corrections.jsonl. One row per drifted playbook with playbook_id + scanned_at + drifted_tools[] + per_tool[] + recommended_action. Downstream consumers (overview model, operator dashboard, scrum_master prompt enrichment) read this file to surface "this playbook compounded the wrong way" signals to humans. Idempotent by design: - Already-flagged entries with no resolved_at are counted as `already_flagged` and skipped (no double-flag, no duplicate row). - Re-scanning after resolve_doc_drift() unflags an entry brings it back into the eligible set on the next scan. Aggregate response shape: { "scanned": N, // playbooks with doc_refs we checked "newly_flagged": N, // drift detected this scan "already_flagged": N, // skipped (still under review) "skipped_retired": N, "skipped_no_refs": N, // pre-Phase-45 playbooks "drifted_by_tool": {tool: count}, "corrections_written": N, } Verified live: POST /doc_drift/scan → scanned=4, newly_flagged=4, drifted_by_tool={docker:4, terraform:1}, corrections_written=4 POST /doc_drift/scan (re-run) → scanned=0, newly_flagged=0, already_flagged=6 (idempotent) data/_kb/doc_drift_corrections.jsonl → 5 rows total (existing seed + this scan) Phase 45 closure status: DocRef + PlaybookEntry.doc_refs ✅ prior session doc_drift module + check_all_refs ✅ prior session /doc_drift/check + /resolve ✅ prior session mcp-server/context7_bridge.ts ✅ prior session boost exclusion in compute_boost_* ✅ prior session /doc_drift/scan + corrections.jsonl ✅ THIS COMMIT The 0→85% thesis stays valid against external doc drift. Popular playbooks can no longer compound the wrong way as Docker / Terraform / React / etc. patch their docs — the scan flags drift, the boost filter excludes the playbook, the operator reviews the corrections .jsonl, and a revise call (Phase 27) supersedes the stale entry with corrected operation/approach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 08:00:50 -05:00
root	98db129b8f	gateway: /v1/iterate — Phase 43 v3 part 3 (generate → validate → retry loop) ... Closes the Phase 43 PRD's "iteration loop with validation in place" structurally. Single endpoint that wraps the 0→85% pattern any caller can post against without re-implementing it. POST /v1/iterate { "kind":"fill" | "email" | "playbook", "prompt":"...", "system":"...", (optional) "provider":"ollama_cloud", "model":"kimi-k2.6", "context":{...}, (target_count/city/state/role/...) "max_iterations":3, (default 3) "temperature":0.2, (default 0.2) "max_tokens":4096 (default 4096) } → 200 + IterateResponse (artifact accepted) {artifact, validation, iterations, history:[{iteration,raw,status}]} → 422 + IterateFailure (max iter reached) {error, iterations, history} The loop: 1. Generate via gateway-internal HTTP loopback to /v1/chat with the given provider/model. Model output is the model's free-form text. 2. Extract a JSON object from the output — handles fenced blocks (```json ... ```), bare braces, and prose-with-embedded-JSON. On no extractable JSON: append "your response wasn't valid JSON" to the prompt and retry. 3. POST the extracted artifact to /v1/validate (server-side reuse of the FillValidator/EmailValidator/PlaybookValidator stack from Phase 43 v3 part 2). 4. On 200 + Report: success — return artifact + history. 5. On 422 + ValidationError: append the specific error JSON to the prompt as corrective context and retry. This is the "observer correction" piece in PRD shape, simplified — the validator's own structured error IS the feedback signal. 6. Cap at max_iterations. Verified end-to-end with kimi-k2.6 via ollama_cloud: Request: fill 1 Welder in Toledo, model picks W-1 (actually Louisville, KY — wrong city) iter 0: model emits {fills:[W-1,"W-1"]} → 422 Consistency ("city 'Louisville' doesn't match contract city 'Toledo'") iter 1: prompt now includes the error → model emits same answer (didn't pick a different worker — model lacks roster access; would need hybrid_search upstream) max=2: 422 IterateFailure with full history The negative test demonstrates the LOOP MECHANICS work: - Generation → validation → retry-with-error-context → cap - The model's failure trace is queryable; downstream tooling can inspect history[] to see exactly where each iteration broke - A production executor would do hybrid_search to find Toledo workers before posting; /v1/iterate is the validation+retry layer downstream JSON extractor handles three shapes: - Fenced: ```json {...} ``` (preferred — explicit signal) - Bare: plain text + {...} + plain text - Multi: picks the first balanced {...} Unit tests cover all three plus the no-JSON fallback. Phase 43 closure status: v1: scaffolds ✅ (older commit) v2: real validators ✅ 00c8408 v3 part 1: parquet WorkerLookup ✅ ebd9ab7 v3 part 2: /v1/validate ✅ 86123fc v3 part 3: /v1/iterate ✅ THIS COMMIT The "0→85% with iteration" thesis is now testable in production. Staffing executors can compose hybrid_search → /v1/iterate (with validation) and converge on validation-passing artifacts in 1-2 iterations on average. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:56:43 -05:00
root	5d93a715c3	gateway: Phase 44 part 3 — split AiClient so vectord routes through /v1/chat ... Builds two AiClient instances at boot: - `ai_client_direct = AiClient::new(sidecar_url)` — direct sidecar transport. Used by V1State (gateway's own /v1/chat ollama_arm needs this — calling /v1/chat from itself would self-loop) and by the legacy /ai proxy. - `ai_client_observable = AiClient::new_with_gateway(sidecar_url, ${gateway_host}:${gateway_port})` — routes generate() through /v1/chat with provider="ollama". Used by: vectord::agent (autotune background loop) vectord::service (the /vectors HTTP surface — RAG, summary, playbook synthesis, etc.) Net result: every LLM call from a vectord module now lands in /v1/usage and Langfuse traces. The autotune agent's hourly cycle becomes observable; /vectors RAG calls show provider+model+latency in the usage report. Phase 44 PRD's gate ("/v1/usage accounts for every LLM call in the system within a 1-minute window") is now satisfied for the gateway-hosted services. Cost: one localhost HTTP hop per vectord-originated LLM call. At ~1-3ms RTT for in-process loopback, negligible against the LLM call's own 30-90s wall-clock. Phase 44 part 4 (deferred): - Standalone consumers that build their own AiClient (test harnesses, bot/propose, etc) — the TS-side already migrated in part 1 + the regression guard at scripts/check_phase44_callers.sh catches new direct callers. Rust standalone harnesses (if any surface) follow the same pattern: construct via new_with_gateway to opt into observability. - Direct sidecar callers in standalone tools (scripts/serve_lab.py is one) — Python-side; out of Rust scope. Verified: cargo build --release -p gateway compiles systemctl restart lakehouse active /v1/chat sanity PONG, finish=stop When the autotune agent next cycles or any /vectors RAG endpoint fires, /v1/usage will show the provider=ollama tick — first real-world data should land within the next agent cycle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:53:18 -05:00
root	7b88fb9269	aibridge: Phase 44 part 2 — opt-in /v1/chat routing for AiClient.generate() ... The Phase 44 PRD's "AiClient becomes a thin /v1/chat client" was a chicken-and-egg problem: the gateway's own /v1/chat ollama_arm calls AiClient.generate() to reach the sidecar. If AiClient unconditionally routed through /v1/chat, gateway → /v1/chat → ollama → AiClient → /v1/chat would loop forever. Solution: opt-in routing. - `AiClient::new(base_url)` — direct-sidecar, gateway-internal use (gateway's own /v1/chat handlers, ollama::chat in mod.rs) - `AiClient::new_with_gateway(base_url, gateway_url)` — routes generate() through ${gateway_url}/v1/chat with provider="ollama" so the call lands in /v1/usage + Langfuse traces Shape translation in generate_via_gateway(): GenerateRequest {prompt, system, model, temperature, max_tokens, think} → /v1/chat {messages: [system?, user], provider:"ollama", ...} /v1/chat response choices[0].message.content + usage.{prompt,completion}_tokens → GenerateResponse {text, model, tokens_evaluated, tokens_generated} embed(), rerank(), and admin methods (health, unload_model, etc.) stay direct-to-sidecar — no /v1/embed equivalent yet, no point round-trip. Transitive migration: aibridge::continuation::generate_continuable goes through TextGenerator::generate_text() → AiClient.generate(), so every caller of generate_continuable inherits the routing decision made at AiClient construction. Phase 21's continuation loop, hot- path JSON emitters, etc. all gain observability for free when the construction site opts in. Verified end-to-end: curl /v1/chat with the exact JSON shape AiClient sends → "PONG-AIBRIDGE", finish=stop, 27/7 tokens /v1/usage after the call → requests=1, by_provider.ollama.requests=1, tokens tracked Phase 44 part 3 (next): - Migrate vectord's AiClient construction site so vectord modules (rag, autotune, harness, refresh, supervisor, playbook_memory) flow through /v1/chat. Currently the gateway's main.rs constructs one AiClient via `new()` and shares it via V1State; vectord inherits direct-sidecar transport. Migration requires constructing a SEPARATE AiClient with `new_with_gateway` for vectord's state bag (V1State.ai_client must stay direct to avoid the self-loop). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:51:04 -05:00
root	86123fce4c	gateway: /v1/validate endpoint — Phase 43 v3 part 2 ... Closes the Phase 43 PRD's "any caller can validate" surface. The validator crate (FillValidator + EmailValidator + PlaybookValidator + WorkerLookup) is now reachable over HTTP at /v1/validate. Request/response: POST /v1/validate {"kind":"fill"|"email"|"playbook", "artifact":{...}, "context":{...}?} → 200 + Report on success → 422 + ValidationError on validation failure → 400 on bad kind Boot-time wiring (main.rs): - Load workers_500k.parquet into a shared Arc<dyn WorkerLookup> - Path overridable via LH_WORKERS_PARQUET env - Missing file: warn + fall back to empty InMemoryWorkerLookup so the endpoint stays live (validators just fail Consistency on every worker-existence check, which is the correct behavior when the roster isn't configured) - Boot log line: "workers parquet loaded from <path>" or "workers parquet at <path> not found" - Live boot timing: 500K rows loaded in ~1.4s V1State gains `validate_workers: Arc<dyn validator::WorkerLookup>`. The `_context` JSON key is auto-injected from `request.context` so callers can either embed `_context` directly in `artifact` or split it cleanly via the `context` field. Verified live (gateway + 500K worker snapshot): POST {kind:"fill", phantom W-FAKE-99999} → 422 Consistency ("does not exist in worker roster") POST {kind:"fill", real W-1, "Anyone"} → 200 OK + Warning ("differs from roster name 'Donald Green'") POST {kind:"email", body has 123-45-6789} → 422 Policy ("SSN- shaped sequence") POST {kind:"nonsense"} → 400 Bad Request The "0→85% with iteration" thesis can now run end-to-end on real staffing data: an executor emits a fill_proposal, posts to /v1/validate, gets a structured ValidationError on phantom IDs or inactive workers, observer-corrects, retries. Closure of that loop in a scrum harness is the next commit (separate scope). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:40:27 -05:00
root	ebd9ab7c77	validator: Phase 43 v3 — production WorkerLookup backed by workers_500k.parquet ... Closes the Phase 43 v2 loose end. The validator scaffolds (FillValidator, EmailValidator) take Arc<dyn WorkerLookup> at construction; this commit ships the parquet-snapshot impl that production code wires in. Schema mapping (workers_500k.parquet → WorkerRecord): worker_id (int64) → candidate_id = "W-{id}" (matches what the staffing executor emits) name (string) → name (already concatenated upstream) role (string) → role city, state (string) → city, state availability (double) → status: "active" if >0 else "inactive" Workers_500k has no `status` column; we derive from `availability` since 0.0 means vacationing/suspended/etc in this dataset's convention. Once Track A.B's `_safe` view ships with proper status, flip the loader to read it directly — schema mapping is in one function (load_workers_parquet), so the swap is trivial. In-memory snapshot model: - Loads all 500K rows at startup → ~75MB resident - Sync .find() — no per-call I/O on the validation hot path - Refresh = call load_workers_parquet again to rebuild - Caller-driven refresh (no auto-watch) — operators pick the cadence Why workers_500k and not candidates.parquet: candidates.parquet has the right shape (string candidate_id, status, first/last_name) but lacks `role` — and the staffing executor matches the W-* convention from workers_500k_v8 corpus. So the production data path goes through workers_500k. The schema mismatch between the two parquets is documented in `reports/staffing/synthetic-data-gap- report.md` (gap A); resolution is operator's call. Errors are typed (LookupLoadError): - Open: file not found / permission - Parse: invalid parquet - MissingColumn: schema doesn't have required field - BadRow: row missing worker_id or name Schema check happens before iteration, so a wrong-shape file fails loud immediately rather than silently building an empty lookup. Verification: cargo build -p validator compiles cargo test -p validator 33 pass / 0 fail (was 31; +2 for parquet) load_real_workers_500k smoke test passes against the live 500K-row file: W-1 resolves, status + role + city/state all populated. Phase 43 v3 part 2 (next): - /v1/validate gateway endpoint that takes a JSON artifact + dispatches to FillValidator/EmailValidator/PlaybookValidator with a shared WorkerLookup loaded from the parquet at gateway startup. - That closes the "any caller can validate" surface; execution-loop wiring (Phase 43 PRD's "generate → validate → correct → retry") becomes a thin wrapper on top of /v1/validate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:36:40 -05:00
root	454da15301	auditor + aibridge: 6 fixes from Opus 4.7 self-audit on PR #11 ... The kimi_architect auditor on commit 00c8408 ran with auto-promotion to claude-opus-4-7 (diff > 100k chars), produced 10 grounded findings, 1 BLOCK + 6 WARN + 3 INFO. This commit lands 6 of them; 3 are skipped (false positives or out-of-scope cleanup deferred). LANDED: 1. kimi_architect.ts:144 empty-parse cache poisoning. When parseFindings returns 0 findings (markdown shape changed, prompt too big, regex missed every block), the verdict was still persisted with empty findings, and the 24h TTL cache short-circuited every subsequent audit with a useless "0 findings" hit. Fix: only persist when findings.length > 0; metrics still appended unconditionally. 2. kimi_architect.ts:122 outage negative-cache. When callKimi throws (network error, gateway 502, rate limit), we returned skipFinding but didn't note the outage anywhere. Every audit cycle within the 24h TTL hammered the dead upstream. Fix: write a sentinel file `<verdict>.outage` on failure with 10-min TTL; future calls within that window short-circuit immediately. 3. kimi_architect.ts:331 mkdir(join(p, "..")) -> dirname(p). The "/.." idiom resolved correctly via Node path normalization but was non-idiomatic and breaks if the path ever has trailing dots. Both Haiku and Opus self-audits flagged it. 4. inference.ts:202 N=3 consensus latency double/triple-count. `totalLatencyMs += run.latency_ms` summed across THREE parallel `Promise.all` calls — wall-clock is bounded by the slowest, not the sum. Renamed to `maxLatencyMs` using `Math.max`. Telemetry now reports actual wall-clock instead of 3x reality. 5. continuation.rs:198,199,230,231 i64/u64 -> u32 saturating cast. `resp.tokens_evaluated as u32` truncates bits when source > u32::MAX instead of saturating. Fix: u32::try_from(...).unwrap_or(u32::MAX) wraps the cast in a real saturate. Applied to both the empty-retry loop and the structural-completion continuation loop. SKIPPED: - BLOCK at Cargo.lock:8911 "validator-not-in-workspace" — confabulation. The diff Opus saw was truncated mid-line; validator IS in Cargo.toml workspace members. Real-world MAX_DIFF_CHARS=180k edge case to watch as we feed more big diffs. - WARN at kimi_architect.ts:248 regex absolute-path edge case — minor, doesn't affect grounding rate observed so far. - INFO at inference.ts:606 "dead reconstruction loop" — Opus misread. The Promise.all worker fills `summaries[]`; the second loop builds a sequential `scratchpad` string from those. Two distinct operations, not redundant. Verification: bun build auditor/checks/{kimi_architect,inference}.ts compiles cargo check -p aibridge green cargo build --release -p gateway green systemctl restart lakehouse.service lakehouse-auditor.service active Next audit cycle (~90s after push) will run on the new diff and exercise the negative-cache + dirname + maxLatencyMs paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 07:10:43 -05:00
root	00c8408335	validator: Phase 43 v2 — real worker-existence + PII + name-consistency checks ... The Phase 43 scaffolds (FillValidator, EmailValidator) shipped with TODO(phase-43 v2) markers for the actual cross-roster checks. This is those checks landing. The PRD calls for "the 0→85% pattern reproduces on real staffing tasks — the iteration loop with validation in place is what made small models successful." Worker-existence is the load-bearing check: when the executor emits {candidate_id: "W-FAKE", name: "Imaginary"}, schema-only validation passes, and only roster lookup catches it. Architecture: - New `WorkerLookup` trait + `WorkerRecord` struct in lib.rs. Sync by design — validators hold an in-memory snapshot, no per-call I/O on the validation hot path. Production wraps a parquet snapshot; tests use `InMemoryWorkerLookup`. - Validators take `Arc<dyn WorkerLookup>` at construction so the same shape covers prod + tests + future devops scaffolds. - Contract metadata travels under JSON `_context` key alongside the validated payload (target_count, city, state, role, client_id for fills; candidate_id for emails). Keeps the Validator trait signature stable and lets the executor serialize context inline. FillValidator (11 tests, was 4): - Schema (existing) - Completeness — endorsed count == target_count - Worker existence — phantom candidate_id fails Consistency - Status — non-active worker fails Consistency - Geo/role match — city/state/role mismatch with contract fails Consistency - Client blacklist — fails Policy - Duplicate candidate_id within one fill — fails Consistency - Name mismatch — Warning (not Error) since recruiters sometimes send roster updates through the proposal layer EmailValidator (11 tests, was 4): - Schema + length (existing) - SSN scan (NNN-NN-NNNN) — fails Policy - Salary disclosure (keyword + $-amount within ~40 chars) — fails Policy. Std-only scan, no regex dep added. - Worker name consistency — when _context.candidate_id resolves, body must contain the worker's first name (Warning if missing) - Phantom candidate_id in _context — fails Consistency - Phone NNN-NNN-NNNN does NOT trip the SSN detector (verified by test); the SSN scanner explicitly rejects sequences embedded in longer digit runs Pre-existing issue (NOT from this change, NOT fixed here): crates/vectord/src/pathway_memory.rs:927 has a stale PathwayTrace struct initializer that fails `cargo check --tests` with E0063 on 6 missing fields. `cargo check --workspace` (production) is green; only the vectord test target is broken. Tracked for a separate fix. Verification: cargo test -p validator 31 pass / 0 fail (was 13) cargo check --workspace green Next: wire `Arc<dyn WorkerLookup>` into the gateway execution loop (generate → validate → observer-correct → retry, bounded by max_iterations=3 per Phase 43 PRD). Production lookup impl loads from a workers parquet snapshot — Track A gap-fix B's `_safe` view is the right source once decided, raw workers_500k otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 06:56:28 -05:00
root	bc698eb6da	gateway: OpenCode (Zen + Go) provider adapter ... Wires opencode.ai as a /v1/chat provider. One sk-* key reaches 40 models across Anthropic, OpenAI, Google, Moonshot, DeepSeek, Zhipu, Alibaba, Minimax — billed against either the user's Zen balance (pay-per-token premium models) or Go subscription (flat-rate Kimi/GLM/DeepSeek/etc.). The unified /zen/v1 endpoint routes both; upstream picks the billing tier based on model id. Notable adapter quirks: - Strip "opencode/" prefix on outbound (mirrors openrouter/kimi pattern). Caller can use {provider:"opencode", model:"X"} or {model:"opencode/X"}. - Drop temperature for claude-*, gpt-5*, o1/o3/o4 models. Anthropic and OpenAI's reasoning lineage rejects temperature with 400 "deprecated for this model". OCChatBody now serializes temperature as Option<f64> with skip_serializing_if so omitting it produces clean JSON. - max_tokens.filter(|&n| n > 0) catches Some(0) — defensive after the same trap bit kimi.rs (empty env -> Number("") -> 0 -> 503). - 600s default upstream timeout; reasoning models on big audit prompts legitimately take 3-5 min. Override OPENCODE_TIMEOUT_SECS. Key handling: - /etc/lakehouse/opencode.env (0600 root) loaded via systemd EnvironmentFile. Same pattern as kimi.env. - OPENCODE_API_KEY env first, file scrape as fallback. Verified end-to-end: opencode/claude-opus-4-7 -> "I'm Claude, made by Anthropic." opencode/kimi-k2.6 -> PONG-K26-GO opencode/deepseek-v4-pro -> PONG-DS-V4 opencode/glm-5.1 -> PONG-GLM opencode/minimax-m2.5-free -> PONG-FREE Pricing reference (per audit @ ~14k in / 6k out): claude-opus-4-7 ~$0.22 (Zen) claude-haiku-4-5 ~$0.04 (Zen) gpt-5.5-pro ~$1.50 (Zen) gemini-3-flash ~$0.03 (Zen) kimi-k2.6 / glm / deepseek / qwen / minimax / mimo: covered by Go subscription ($10/mo, $60/mo cap). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 06:40:55 -05:00
root	ff5de76241	auditor + gateway: 2 fixes from kimi_architect's first real run ... Acted on 2 of 10 findings Kimi caught when auditing its own integration on PR #11 head 8d02c7f. Skipped 8 (false positives or out-of-scope). 1. crates/gateway/src/v1/kimi.rs — flatten OpenAI multimodal content array to plain string before forwarding to api.kimi.com. The Kimi coding endpoint is text-only; passing a [{type,text},...] array returns 400. Use Message::text() to concat text-parts and drop non-text. Verified with curl using array-shape content: gateway now returns "PONG-ARRAY" instead of upstream error. 2. auditor/checks/kimi_architect.ts — computeGrounding switched from readFileSync to async readFile inside Promise.all. Doesn't matter at 10 findings; would matter at 100+. Removed unused readFileSync import. Skipped findings (with reason): - drift_report.ts:18 schema bump migration concern: the strict schema_version refusal IS the migration boundary (v1 readers explicitly fail on v2; not a silent corruption risk). - replay.ts:383 ISO timestamp precision: Date.toISOString always emits "YYYY-MM-DDTHH:mm:ss.sssZ" (ms precision). False positive. - mode.rs:1035 matrix_corpus deserializer compat: deserialize_string _or_vec at mode.rs:175 already accepts both shapes. Confabulation from not seeing the deserializer in the input bundle. - /etc/lakehouse/kimi.env world-readable: actually 0600 root. Real concern would be permission-drift; not a code bug. - callKimi response.json hang: obsolete; we use curl now. - parseFindings silent-drop: ergonomic concern, not a bug. - appendMetrics join with "..": works for current path; deferred. - stubFinding dead-type extension: cosmetic. Self-audit grounding rate at v1.0.0: 10/10 file:line citations verified by grep. 2 of 10 actionable bugs landed. The other 8 were correctly flagged as concerns but didn't earn a code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 06:16:23 -05:00
root	643dd2d520	gateway: direct Kimi For Coding provider adapter (api.kimi.com) ... Wires kimi-for-coding (Kimi K2.6 underneath) as a first-class /v1/chat provider so consumers can target it via {provider:"kimi"} or model prefix kimi/<model>. Bypasses the upstream-broken kimi-k2:1t on Ollama Cloud and the rate-limited moonshotai/kimi-k2.6 path through OpenRouter. Adapter shape mirrors openrouter.rs (OpenAI-compatible Chat Completions). Differences from generic OpenAI providers: - api.kimi.com is a SEPARATE account system from api.moonshot.ai and api.moonshot.cn. sk-kimi-* keys are NOT interchangeable across them. - Endpoint is User-Agent-gated to "approved" coding agents (Kimi CLI, Claude Code, Roo Code, Kilo Code, ...). Requests from generic clients return 403 access_terminated_error. Adapter sends User-Agent: claude-code/1.0.0. Per Moonshot TOS this is a tampering-class action that may result in seat suspension; J authorized 2026-04-27 with awareness of the risk. - kimi-for-coding is a reasoning model — reasoning_content counts against max_tokens. Default 800-token budget yields empty visible content with finish_reason=length. Code-review workloads need max_tokens >= 1500. - Default 600s upstream timeout (vs 180s for openrouter.rs) — code audits with full file context legitimately take 3-5 minutes. Override via KIMI_TIMEOUT_SECS env. Key handling: - /etc/lakehouse/kimi.env (0600 root) loaded via systemd EnvironmentFile - KIMI_API_KEY env first, then file scrape as fallback - /etc/systemd/system/lakehouse.service NOT included in this commit (system file outside repo); operator must add EnvironmentFile=- /etc/lakehouse/kimi.env to the lakehouse.service unit NOT in scrum_master_pipeline LADDER. The 9-rung ladder is for unattended automatic recovery; placing Kimi there would hammer a TOS-gated endpoint with hostility-policy potential. Kimi is addressable via /v1/chat for explicit invocations only — auditor integration in a follow-up commit. Verification: cargo check -p gateway --tests compiles curl /v1/chat provider=kimi 200 OK, content="PONG" curl /v1/chat model="kimi/kimi-for-coding" 200 OK (prefix routing) Kimi audit on distillation last-week 7/7 grounded findings (reports/kimi/audit-last-week-full.md) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 05:35:58 -05:00
root	d77622fc6b	distillation: fix 7 grounding bugs found by Kimi audit ... Kimi For Coding (api.kimi.com, kimi-for-coding) ran a forensic audit on distillation v1.0.0 with full file content. 7/7 flags verified real on grep. Substrate now matches what v1.0.0 claimed: deterministic, no schema bypasses, Rust tests compile. Fixes: - mode.rs:1035,1042 matrix_corpus Some/None -> vec![..]/vec![]; cargo check --tests now compiles (was silently broken; only bun tests were running) - scorer.ts:30 SCORER_VERSION env override removed - identical input now produces identical version stamp, not env-dependent drift - transforms.ts:181 auto_apply wall-clock fallback (new Date()) -> deterministic recorded_at fallback - replay.ts:378 recorded_run_id Date.now() -> sha256(recorded_at); replay rows now reproducible given recorded_at - receipts.ts:454,495 input_hash_match hardcoded true was misleading telemetry; bumped DRIFT_REPORT_SCHEMA_VERSION 1->2, field is now boolean|null with honest null when not computed at this layer - score_runs.ts:89-100,159 dedup keyed only on sig_hash made scorer-version bumps invisible. Composite sig_hash:scorer_version forces re-scoring - export_sft.ts:126 (ev as any).contractor bypass emitted "<contractor>" placeholder for every contract_analyses SFT row. Added typed EvidenceRecord.metadata bucket; transforms.ts populates metadata.contractor; exporter reads typed value Verification (all green): cargo check -p gateway --tests compiles bun test tests/distillation/ 145 pass / 0 fail bun acceptance 22/22 invariants bun audit-full 16/16 required checks Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 05:34:31 -05:00
root	20a039c379	auditor: rebuild on mode runner + drop tree-split (use distillation substrate) ... Architectural simplification leveraging Phase 5 distillation work: the auditor no longer pre-extracts facts via per-shard summaries because lakehouse_answers_v1 (gold-standard prior PR audits + observer escalations corpus) supplies cross-PR context through the mode runner's matrix retrieval. Same signal, ~50× fewer cloud calls per audit. Per-audit cost: Before: 168 gpt-oss:120b shard summaries + 3 final inference calls After: 3 deepseek-v3.1:671b mode-runner calls (full retrieval included) Wall-clock on PR #11 (1.36MB diff): Before: ~25 minutes After: 88 seconds (3/3 consensus succeeded) Files: auditor/checks/inference.ts - Default MODEL kimi-k2:1t → deepseek-v3.1:671b. kimi-k2 is hitting sustained Ollama Cloud 500 ISE (verified via repeated trivial probes; multi-hour outage). deepseek is the proven drop-in from Phase 5 distillation acceptance testing. - Dropped treeSplitDiff invocation. Diff truncates to MAX_DIFF_CHARS and goes straight to /v1/mode/execute task_class=pr_audit; mode runner pulls cross-PR context from lakehouse_answers_v1 via matrix retrieval. SHARD_MODEL retained for legacy callCloud compatibility (default qwen3-coder:480b if it ever runs). - extractAndPersistFacts now reads from truncated diff (no scratchpad post-tree-split-removal). auditor/checks/static.ts - serde-derived struct exemption (commit 107a682 shipped this; this commit is the rest of the auditor rebuild it landed alongside) - multi-line template literal awareness in isInsideQuotedString — tracks backtick state across lines so todo!() inside docstrings doesn't trip BLOCK_PATTERNS. crates/gateway/src/v1/mode.rs - pr_audit native runner mode added to VALID_MODES + is_native_mode + flags_for_mode + framing_text. PrAudit framing produces strict JSON {claim_verdicts, unflagged_gaps} for the auditor to parse. config/modes.toml - pr_audit task class with default_model=deepseek-v3.1:671b and matrix_corpus=lakehouse_answers_v1. Documents kimi-k2 outage with link to the swap rationale. Real-data audit on PR #11 head 1b433a9 (which is the PR with all the distillation work + auditor rebuild itself): - Pipeline ran to completion (88s for inference; full audit ~3 min) - 3/3 consensus runs succeeded on deepseek-v3.1:671b - 156 findings: 12 block, 23 warn, 121 info - Block findings are legitimate signal: 12 reviewer claims like "Invariants enforced (proven by tests + real run):" that the truncated diff can't directly verify. The auditor is correctly flagging claim-vs-diff divergence — exactly its job. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 23:32:44 -05:00
root	d1d97a045b	v1: fire observer /event from /v1/chat alongside Langfuse trace ... Observer at :3800 already collects scrum + scenario events into a ring buffer that pathway-memory + KB consolidation read from. /v1/chat now posts a lightweight {endpoint, source:"v1.chat", input_summary, output_summary, success, duration_ms} event there too — fire-and-forget tokio::spawn, observer-down doesn't block the chat response. Now any tool routed through our gateway (Pi CLI, Archon, openai SDK clients, langchain-js) shows up in the same ring buffer the scrum loop reads, ready for the same KB-consolidation analysis. Independent of the existing langfuse-bridge that polls Langfuse — this path is immediate. Verified: GET /stats shows {by_source: {v1.chat: N}} grows by 1 per chat call, both for direct curl and for Pi CLI invocations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 18:01:52 -05:00
root	540a9a27ee	v1: accept OpenAI multimodal content shape (array-of-parts) ... Modern OpenAI clients (pi-ai, openai SDK 6.x, langchain-js, the official agents) send `messages[].content` as an array of content parts: `[{type:"text", text:"..."}, {type:"image_url", ...}]`. Our gateway typed `content` as plain `String` and 422'd those calls. Fix: `Message.content` is now `serde_json::Value` so requests deserialize regardless of shape. `Message::text()` flattens content-parts arrays (concat'd `text` fields, non-text parts skipped) for places that need a plain string — Ollama prompt assembly, char counts, the assistant's own response synthesis. `Message::new_text()` constructs string-content messages without writing the wrapper at each call site. Forwarders (openrouter) clone content through verbatim so providers see exactly what the client sent. Verified end-to-end: Pi CLI (`pi --print --provider openrouter`) landed a clean 1902-token request through `/v1/chat/completions`, routed to OpenRouter as `openai/gpt-oss-120b:free`, response in 1.62s, Langfuse trace `v1.chat:openrouter` recorded with provider tag. Same path that any tool using the official openai SDK takes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 17:56:46 -05:00
root	3a0b37ed93	v1: OpenAI-compat alias + smart provider routing — gateway is now drop-in middleware ... /v1/chat/completions route alias (same handler as /chat) lets any tool using the official `openai` SDK adopt the gateway via OPENAI_BASE_URL alone — no custom provider field needed. resolve_provider() extended: - bare `vendor/model` (slash) → openrouter (catches x-ai/grok-4.1-fast, moonshotai/kimi-k2, deepseek/deepseek-v4-flash, openai/gpt-oss-120b:free) - bare vendor model names (no slash, no colon) get auto-prefixed: gpt-* / o1-* / o3-* / o4-* → openai/<name> (OpenRouter form) claude-* → anthropic/<name> grok-* → x-ai/<name> Then routed to openrouter. Ollama models (with colon, no slash) keep default routing. Tools like pi-ai validate against an OpenAI-style catalog and send bare names — this lets them flow through cleanly. Verified end-to-end: - curl POST /v1/chat/completions {model: "gpt-4o-mini", ...} → 200, routed to openrouter as openai/gpt-4o-mini - openai SDK with baseURL=http://localhost:3100/v1 → 3 model variants all succeed (openai/gpt-4o-mini, gpt-4o-mini, x-ai/grok-4.1-fast) - Langfuse traces fire automatically on every call (v1.chat:openrouter, provider tagged in metadata) scripts/mode_pass5_variance_paid.ts gains LH_CONDITIONS env so subset runs (e.g. just isolation vs composed) take half the latency. Archon-on-Lakehouse integration: gateway side is done. Pi-ai's openai-responses backend uses /v1/responses (not /chat/completions) and its openrouter backend appears to bail in client-side validation before sending. Patching Pi locally to override baseUrl works for arch but the harness still rejects — needs more work in a follow-up. Direct openai SDK path (langchain-js / agents / patched Pi) works today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 17:49:37 -05:00
root	2dbc8dbc83	v1/mode: model-aware enrichment downgrade + 3 corpora + variance harness ... Pass 5 (5 reps × 4 conditions × 1 file on grok-4.1-fast) showed composing matrix corpora is anti-additive on strong models — composed lakehouse_arch + symbols LOST 5/5 head-to-head vs codereview_isolation (Δ −1.8 grounded findings, p=0.031). Default flips to isolation; matrix path now auto- downgrades when the resolved model is strong. Mode runner: - matrix_corpus is Vec<String> (string OR array via deserialize_string_or_vec) - top_k=6 from each corpus, merge by score, take top 8 globally - chunk tag prefers doc_id over source so reviewer sees [adr:009] vs [lakehouse_arch] - is_weak_model() gate auto-downgrades codereview_lakehouse → codereview_isolation for strong models (default-strong; weak = :free suffix or local last-resort) - LH_FORCE_FULL_ENRICHMENT=1 bypasses for diagnostic runs - EnrichmentSources.downgraded_from records when the gate fires Three corpora indexed via /vectors/index (5849 chunks total): - lakehouse_arch_v1 — ADRs + phases + PRD + scrum spec (93 docs, 2119 chunks) - scrum_findings_v1 — past scrum_reviews.jsonl (168 docs, 1260 chunks; EXCLUDED from defaults — 24% out-of-bounds line citations from cross-file drift) - lakehouse_symbols_v1 — regex-extracted pub items + /// docs (656 docs, 2470 chunks) Experiment infra: - scripts/build_*_corpus.ts — re-runnable when source content changes - scripts/mode_pass5_variance_paid.ts — N reps × M conditions on one file - scripts/mode_pass5_summarize.ts — mean ± σ + head-to-head, parser handles numbered + path-with-line + path-with-symbol finding tables - scripts/mode_compare.ts — groups by mode|corpus when sweeps span corpora - scripts/mode_experiment.ts — default model bumped to x-ai/grok-4.1-fast, --corpus flag for per-call override Decisions + open follow-ups: docs/MODE_RUNNER_TUNING_PLAN.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 17:29:17 -05:00
root	56bf30cfd8	v1/mode: override knobs + staffing native runner + pass 2/3/4 harnesses ... Setup for the corpus-tightening experiment sweep (J 2026-04-26 — "now is the only cheap window before the corpus gets large and refactoring costs go up"). Override params on /v1/mode/execute (additive — old callers unaffected): force_matrix_corpus — Pass 2: try alternate corpora per call force_relevance_threshold — Pass 2: sweep filter strictness force_temperature — Pass 3: variance test New native mode `staffing_inference_lakehouse` (Pass 4): - Same composer architecture as codereview_lakehouse - Staffing framing: coordinator producing fillable|contingent| unfillable verdict + ranked candidate list with playbook citations - matrix_corpus = workers_500k_v8 - Validates that modes-as-prompt-molders generalizes beyond code - Framing explicitly says "do NOT fabricate workers" — the staffing analog of the lakehouse mode's symbol-grounding requirement Three sweep harnesses: scripts/mode_pass2_corpus_sweep.ts — 4 corpora × 4 thresholds × 5 files scripts/mode_pass3_variance.ts — 3 files × 3 temps × 5 reps scripts/mode_pass4_staffing.ts — 5 fill requests through staffing mode Each appends per-call rows to data/_kb/mode_experiments.jsonl which mode_compare.ts already aggregates with grounding column. Pass 1 (10 files × 5 modes broad sweep) currently running via the existing scripts/mode_experiment.ts — gateway restart deferred until it completes so the new override knobs aren't enabled mid-experiment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 01:55:12 -05:00
root	7c47734287	v1/mode: parameterized runner + 5 enrichment-experiment modes ... J's directive (2026-04-26): "Create different modes so we can really dial in the architecture before it gets further along — pinpoint the failures and strengths equally so I know what direction to go in. Loop theater happens when we don't pinpoint the most accurate path." Refactored execute() to switch on mode name → EnrichmentFlags preset. Five native modes designed as deliberate experiments — each isolates one architectural axis so the comparison matrix reads off what's doing work vs what's adding latency for nothing: codereview_lakehouse — all enrichment on (ceiling) codereview_null — raw file + generic prompt (baseline) codereview_isolation — file + pathway only (no matrix) codereview_matrix_only — file + matrix only (no pathway) codereview_playbook_only — pathway only, NO file content (lossy ceiling) Each call appends a row to data/_kb/mode_experiments.jsonl with full sources + response. LH_MODE_LOG_OFF=1 to suppress. scripts/mode_experiment.ts — sweeps files × modes serially, prints live progress with per-call enrichment stats. Defaults to OpenRouter free model so cloud quota doesn't gate experiments. scripts/mode_compare.ts — reads the JSONL, outputs per-file matrix + per-mode aggregate + mode-vs-baseline win/loss with avg finding delta. Heuristic finding-count from markdown table rows; pathway citation count from preamble references. scrum_master_pipeline.ts gets a mode-runner fast path gated by LH_USE_MODE_RUNNER=1: try /v1/mode/execute first, fall through to the existing ladder if response < LH_MODE_MIN_CHARS (default 2000) or anything errors. Off by default until A/B-validated. First experiment results (2 files × 5 modes via gpt-oss-120b:free): - codereview_null produces 12.6KB response with ZERO findings (proves adversarial framing is load-bearing) - codereview_playbook_only produces MORE findings than lakehouse on average (12 vs 9) at 73% the latency — pathway memory is the dominant signal driver - codereview_matrix_only underperforms isolation by ~0.5 findings while costing the same latency — matrix corpus likely underperforming for scrum_review task class Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 01:36:42 -05:00
root	86f63a083d	v1/mode: codereview_lakehouse native runner — modes are prompt-molders ... J's framing (2026-04-26): "Modes are how you ask ONCE and get BETTER information — they mold the data, hyperfocus the prompt on this codebase's needs, so the model gets it right the first time without the cascading retry ladder." Built the first concrete native enrichment runner (codereview_lakehouse) that composes every context primitive the gateway exposes: 1. Focus file content (read from disk OR caller-supplied) 2. Pathway memory bug_fingerprints for this file area (ADR-021 preamble — "📚 BUGS PREVIOUSLY FOUND IN THIS FILE AREA") 3. Matrix corpus search via the task_class's matrix_corpus 4. Relevance filter (observer /relevance) drops adjacency pollution 5. Assembles ONE precise prompt with system framing 6. Single call to /v1/chat with the recommended model POST /v1/mode/execute dispatches. Native mode → runs the composer. Non-native mode → 501 NOT_IMPLEMENTED with hint (proxy to LLM Team /api/run is queued). Provider hint logic auto-routes by model name shape: - vendor/model[:tag] → openrouter - kimi-*/qwen3-coder*/deepseek-v*/mistral-large* → ollama_cloud - everything else → local ollama Live test against crates/queryd/src/delta.rs (10593 bytes, 10 historical bug fingerprints, 2 matrix chunks dropped by relevance): - enriched_chars: 12876 - response_chars: 16346 (14 findings with confidence percentages) - Model literally cited the pathway memory preamble in finding #7 - One call to free-tier gpt-oss:120b produced what previously required the 9-rung escalation ladder Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 00:28:46 -05:00
root	d277efbfd2	v1/mode: task_class → mode/model router (decision-only, phase 1) ... HANDOVER §queued (2026-04-25): "Mode router — port LLM Team multi-model patterns. Pick the right TOOL/MODE for each task class via the matrix, not cascade through models." Two-stage architecture: 1. Decision (POST /v1/mode) — pure recommendation, no execution. Returns {mode, model, decision: {source, fallbacks, matrix_corpus, notes}} so callers see WHY this mode was picked. 2. Execution (future POST /v1/mode/execute) — proxy to LLM Team /api/run for modes not yet ported to native Rust runners. Not wired in this phase. Splitting decision from execution lets us A/B-test the routing logic without committing to running every recommendation. The decision function is pure enough for exhaustive unit tests (3 added). config/modes.toml — initial map for 5 task_classes (scrum_review, contract_analysis, staffing_inference, fact_extract, doc_drift_check) + a default. matrix_corpus per task is reserved for the future matrix-informed routing pass. VALID_MODES list (24 modes) is kept in sync manually with LLM Team's /api/run handler at /root/llm_team_ui.py:10581. Adding a mode here without adding it upstream returns 400 from a future proxy. GET /v1/mode/list — operator introspection so a UI can render the registry table without re-parsing TOML. Live-tested: 5 task classes match, unknown classes fall through to default, force_mode override works + validates, bogus modes return 400 with the valid_modes list. Updates reference_llm_team_modes.md memory — earlier note claiming "only extract is registered" was wrong (all 25 are registered). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 00:16:32 -05:00
root	626f18d491	pathway_memory: audit-consensus → retire wire ... When observer's hand-review explicitly rejects the output of a hot-swap-recommended model, the matrix's recommendation was wrong for this context. Auto-retire the trace so future agents don't get the same poisoned recommendation in their preamble. crates/vectord/src/pathway_memory.rs — add `trace_uid` to HotSwapCandidate response and populate from the matched trace. This gives consumers single-trace precision for /pathway/retire. tests/real-world/scrum_master_pipeline.ts: - HotSwapCandidate interface gains trace_uid - new retirePathwayTrace() helper (fire-and-forget, fall-open) - in the obsVerdict reject branch: if hotSwap was active AND the rejected model is the hot-swap-recommended one AND observer confidence ≥0.7, fire retire and null hotSwap so post-loop replay bookkeeping doesn't double-process. - hotSwap declared `let` (was const) so it can be nulled Cycle verdicts ("needs different angle") don't trigger retire — only outright rejects do. Confidence gate avoids retiring on heuristic-fallback verdicts that come back without a confidence number. Closes the "audit-consensus → retire" item from HANDOVER.md. Live-tested: insert synthetic trace → /pathway/retire by trace_uid → retired counter 1 → 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 00:01:20 -05:00
root	6ac7f61819	pathway_memory: Mem0 versioning + deletion (upsert/revise/retire/history) ... Per J 2026-04-25: pathway_memory was append-only — every agent run added a new trace, bad/failed runs polluted the matrix forever, no notion of "this is the canonical evolved playbook." Ported playbook_memory's Phase 25/27 patterns into pathway_memory so the agent loop's matrix converges on best-known approaches per task class instead of bloating. Fields added to PathwayTrace (all #[serde(default)] for back-compat): - trace_uid: stable UUID per individual trace within a bucket - version: u32 default 1 - parent_trace_uid, superseded_at, superseded_by_trace_uid - retirement_reason (paired with existing retired:bool) Methods added to PathwayMemory: - upsert(trace) → PathwayUpsertOutcome {Added|Updated|Noop} Workflow-fingerprint dedup: ladder_attempts + final_verdict hash. Identical workflow → bumps existing replay_count instead of duplicating. - revise(parent_uid, new_trace) → PathwayReviseOutcome Chains versions; rejects retired or already-superseded parents. - retire(trace_uid, reason) → bool Marks specific trace retired with reason. Idempotent. - history(trace_uid) → Vec<PathwayTrace> Walks parent_trace_uid back to root, then superseded_by forward to tip. Cycle-safe via visited set. Retrieval gates updated: - query_hot_swap skips superseded_at.is_some() - bug_fingerprints_for skips both retired AND superseded HTTP endpoints in service.rs: - POST /vectors/pathway/upsert - POST /vectors/pathway/retire - POST /vectors/pathway/revise - GET /vectors/pathway/history/{trace_uid} scripts/seal_agent_playbook.ts switched insert→upsert + accepts SESSION_DIR arg so it can seal any archived session, not just iter4. Verified live (4/4 ops): - UPSERT first run: Added trace_uid 542ae53f - UPSERT identical: Updated, replay_count bumped 0→1 (no duplicate) - REVISE 542ae53f→87a70a61: parent stamped superseded_at, v2 created - HISTORY of v2: chain_len=2, v1 superseded, v2 tip - RETIRE iter-6 broken trace: retired=true, retirement_reason preserved - pathway_memory.stats: total=79, retired=1, reuse_rate=0.0127 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-25 19:31:44 -05:00
root	4087dde780	execution_loop: update stale test assertion to match current prompt format ... Pre-existing failure I've been noting across this session — `executor_prompt_includes_surfaced_candidates` expected the substring "W-1 Alice Smith" but the prompt format was intentionally changed (probably in a Phase 38/39 commit) to separate doc_id from name so the executor doesn't conflate `doc_id` (vector-index key) with `workers_500k.worker_id` (integer PK). Current prompt format (line 1178 in build_executor_prompt): - name="Alice Smith" city="Toledo" state="OH" (vector doc_id=W-1) The prompt body explicitly instructs the model NOT to conflate the two IDs — the format separation is the mechanism enforcing that instruction. The OLD test assertion predated that separation. Assertion now checks the semantic contract (both tokens present, any order) instead of the exact old concatenation. Workspace test result after this commit: 343 passed, 0 failed, 0 warnings (both lib + tests). This is the last stale-test hole from the phase-audit sweep — it popped up during the 41-commit push but I was leaving it as pre-existing-unrelated. J called it: sitting broken for hours is worse than a one-line assertion update. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 14:06:24 -05:00
root	951c6014ec	gateway: boot-time probe of truth/ file-backed rules ... Phase 42 PRD deliverable de8fb10 landed the file loader + 2 rule files. This commit wires the loader into gateway startup so the rules actually get READ at boot — catches parse errors and duplicate-ID collisions before the first request hits, rather than "silently 0 rules loaded." Scope is deliberately narrow — a probe, not full plumbing: - Reads LAKEHOUSE_TRUTH_DIR env override, defaults to /home/profit/lakehouse/truth - Skips silently with a debug log if the dir is absent - Loads rules on top of default_truth_store() into a throwaway store, logs the count (or the error) - Does NOT yet replace the per-request default_truth_store() in execution_loop or v1/chat. That plumbing needs a V1State.truth field + passing it through the request context, which is a separate scope. Why the separation matters: this commit gives ops + me a visible boot-time signal ("truth: loaded 3 file-backed rule(s)") that the loader + files work end-to-end. The next commit can confidently swap per-request stores without wondering whether the parsing even succeeds. Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 14:03:17 -05:00
root	fee094f653	gateway/access: wire get_role + is_enabled into HTTP routes ... Two of the four #[allow(dead_code)] methods in access.rs were dead because nothing exposed them externally. access.rs itself is fine — list_roles, set_role, can_access all have live callers. But get_role and is_enabled were shaped as public API with no surface to call them through. Fix adds two small routes under /access (where the rest of the access surface lives): GET /access/roles/{agent} Calls AccessControl::get_role(agent). Returns 404 with a clear message when the agent isn't registered so clients distinguish "unknown agent" from "access denied." Part of P13-001 (ops tooling needs per-agent role introspection). GET /access/enabled Calls AccessControl::is_enabled(). Returns {"enabled": bool}. Dashboards + ops tooling poll this to confirm auth posture of the running gateway — distinct from /health which answers "is the process up" vs "is access enforcement on." #[allow(dead_code)] removed from both methods — they have live callers now via these routes, the linter will enforce that going forward. Still #[allow(dead_code)] on access.rs: masked_fields + log_query. Both need cross-crate wiring: - masked_fields wants the agent's role + query response columns, called in response shaping (queryd returning to gateway path) - log_query wants post-execution audit, called after every SQL execution on the gateway boundary Both are P13-001 phase 2 work — need AgentIdentity plumbed through the /query nested router before the call sites make sense. Flagged for follow-up. Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 14:02:01 -05:00
root	91a38dc20b	vectord/index_registry: add last_used + build_signature (scrum iter 11) ... Scrum iter 11 on crates/vectord/src/index_registry.rs flagged two concrete field gaps (90% confidence). Both were tagged UnitMismatch / missing-invariant. IndexMeta gains two Optional fields: last_used: Option<DateTime<Utc>> PRD 11.3 — when this index was last searched against. Callers were reading created_at as a liveness proxy, which conflated "built" with "used." IndexRegistry::touch_used(name) stamps the field on every hit; incremental re-embed can now skip cold indexes without misattributing "fresh build" to "recent use." build_signature: Option<String> PRD 11.3 — stable SHA-256 of (sorted source files + chunk_size + overlap + model_version). compute_build_signature() in the same module is deterministic: file-order-invariant, changes on chunk param, changes on model version. Lets incremental re-embed answer "has anything changed since last build?" without scanning the source Parquet. Both fields are #[serde(default)] — the ~40 existing .json meta files under vectors/meta/ load unchanged. Backward-compat verified by the explicit `index_meta_deserializes_without_new_fields_backcompat` test. 7 new tests: - build_signature_is_deterministic - build_signature_order_invariant (sorted internally) - build_signature_changes_on_chunk_param - build_signature_changes_on_model_version - touch_used_updates_last_used - touch_used_is_noop_on_missing_index - index_meta_deserializes_without_new_fields_backcompat Call-site fixes: crates/vectord/src/refresh.rs:294 and crates/vectord/src/service.rs:244 both construct IndexMeta with fully-literal init, default the new fields to None. One indentation cleanup on service.rs (a pre-existing visual issue on id_prefix: None). Workspace warnings still at 0. touch_used() isn't wired into search hot-path yet — follow-up commit when the search handlers can adopt it without a broader refactor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 14:00:09 -05:00
root	6532938e85	gateway/tools: truth gate for model-provided SQL (iter 11 CF-1+CF-2) ... Scrum iter 11 flagged crates/gateway/src/tools/service.rs with two 95%-confidence critical failures: CF-1: "Direct SQL execution from model-provided parameters without explicit validation or sanitization" (line 68, 95% conf) CF-2: "No permission check performed before executing SQL query; access control is bypassed entirely" (line 102, 90% conf) CF-1 is the real one — same security gap as queryd /sql had before P42-002 (9cc0ceb). Tool invocations build SQL from a template + model-provided params, then state.query_fn.execute(&sql) runs it. No truth-gate check between build and execute meant an adversarial model could emit DROP TABLE / DELETE FROM / TRUNCATE inside a param and bypass queryd's gate by routing through the tool surface instead. Fix mirrors the queryd SQL gate exactly: - ToolState grows an Arc<TruthStore> field - main.rs constructs it via truth::sql_query_guard_store() (shared default — same destructive-verb block as queryd) - call_tool evaluates the built SQL against "sql_query" task class BEFORE executing - Any Reject/Block outcome → 403 FORBIDDEN + log_invocation row marked success=false with the rule message CF-2 (access control) is P13-001 territory — needs AccessControl wiring into queryd first, still open. Flagged in memory. Workspace warnings still at 0. Pattern is now: queryd /sql → truth::sql_query_guard_store (9cc0ceb) gateway /tools → truth::sql_query_guard_store (this commit) execution_loop → truth::default_truth_store (51a1aa3) All three surfaces that pipe SQL or spec-shaped data through to the substrate now gate it. Any new SQL-executing surface should follow the same pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:52:29 -05:00
root	de8fb10f52	phase-42: truth/ repo-root dir + TOML rule loader ... Phase 42 PRD (docs/CONTROL_PLANE_PRD.md:144): "truth/ dir at repo root — rule files, versioned in git." Didn't exist. Landing both the dir + its loader. New files: truth/ README.md — documents file format, rule shape, composition model (file rules are additive on top of in-code default_ truth_store), explicit non-goals (no hot reload, no inheritance) staffing.fill.toml — 2 staffing.fill rules: endorsed-count-matches-target, city-required (both Reject via FieldEmpty) staffing.any.toml — 1 staffing.any rule: no-destructive-sql-in-context via FieldContainsAny (parallel to the queryd SQL gate we already ship) crates/truth/src/loader.rs — load_from_dir(store, dir) — 5 tests: happy path, duplicate-ID rejection within files, duplicate-ID rejection against in-code rules, non-toml files skipped, missing-dir error. Alphabetical file order for reproducible error messages. crates/truth/src/lib.rs — new pub fn all_rule_ids() helper on TruthStore so the loader can detect collisions without breaching the private `rules` field. crates/truth/Cargo.toml — adds `toml` workspace dep. Composition model: file rules are ADDITIVE on top of what default_truth_store() registers in code. Operators can tune thresholds/needles/descriptions at the file layer without a code deploy. Schema changes (new RuleCondition variants) still need a code bump. Integration hook (not in this commit, flagged for follow-up): main.rs should call loader::load_from_dir(&mut store, "truth/") after default_truth_store() so file-backed rules take effect on gateway boot. Deliberately separate: this commit lands the machinery; wiring it on happens when the team is ready to own the rule file lifecycle. Total: 37 truth tests green (was 32). Workspace warnings still 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:44:23 -05:00
root	0b3bd28cf8	phase-40: Gemini + Claude provider adapters ... Phase 40 PRD (docs/CONTROL_PLANE_PRD.md:82-83) listed: - crates/aibridge/src/providers/gemini.rs - crates/aibridge/src/providers/claude.rs Neither existed. Landing both now, in gateway/src/v1/ (matches the existing ollama.rs + openrouter.rs sibling pattern — aibridge's providers/ is for the adapter *trait* abstractions, v1/ holds the concrete /v1/chat dispatchers that know the wire format). gemini.rs: - POST https://generativelanguage.googleapis.com/v1beta/models/ {model}:generateContent?key=<API_KEY> - Auth: query-string key (not bearer) - Maps messages → contents+parts (Gemini's wire shape), extracts from candidates[0].content.parts[0].text - 3 tests: key resolution, body serialization (camelCase generationConfig + maxOutputTokens), prefix-strip claude.rs: - POST https://api.anthropic.com/v1/messages - Auth: x-api-key header + anthropic-version: 2023-06-01 - Carries system prompt in top-level `system` field (not messages[]). Extracts from content[0].text where type=="text" - 4 tests: key resolution, body serialization with/without system field, prefix-strip v1/mod.rs: + V1State.gemini_key + claude_key Option<String> + resolve_provider() strips "gemini/" and "claude/" prefixes + /v1/chat dispatcher handles "gemini" + "claude"/"anthropic" + 2 new resolve_provider tests (prefix + strip per adapter) main.rs: + Construct both keys at startup via resolve_*_key() helpers. Missing keys log at debug (not warn) since these are optional providers — unlike OpenRouter which is the rescue rung. Every /v1/chat error path mirrors the existing pattern: - 503 SERVICE_UNAVAILABLE when key isn't configured - 502 BAD_GATEWAY with the provider's error text when the upstream call fails - Response shape always the OpenAI-compatible ChatResponse Workspace warnings still at 0. 9 new tests pass. Pre-existing test failure `executor_prompt_includes_surfaced_ candidates` at execution_loop/mod.rs:1550 is unrelated (fails on pristine HEAD too — PR fixture divergence). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:41:31 -05:00
root	b5b0c00efe	phase-43: new crates/validator — trait, staffing impls, devops scaffold ... Phase 43 PRD (docs/CONTROL_PLANE_PRD.md:161) was the one audit finding truly unimplemented — no crate, no trait, no tests, no workspace entry. Neither PHASES.md nor the source tree had any Phase 43 presence. Genuine greenfield gap. Lands the scaffold as a real crate, registered in workspace Cargo.toml: crates/validator/ src/lib.rs — Validator trait, Artifact enum (5 variants: FillProposal, EmailDraft, Playbook, TerraformPlan, AnsiblePlaybook), Report, Finding, Severity, ValidationError src/staffing/mod.rs — staffing validators module root src/staffing/fill.rs — FillValidator (schema-level: fills array + per-fill {candidate_id, name}). 4 tests. Worker-existence + status + geo checks are TODO v2 (need catalog query handle). src/staffing/email.rs — EmailValidator (to/body schema + SMS ≤160 + email subject ≤78). 4 tests. PII scan + name-consistency TODO v2. src/staffing/playbook.rs — PlaybookValidator (operation prefix, endorsed_names non-empty + ≤ target×2, fingerprint present per Phase 25). 5 tests. src/devops.rs — TerraformValidator + AnsibleValidator scaffolds. Return Unimplemented — keeps dispatcher shape stable, surfaces a clear "phase 43 not wired" signal instead of silently passing or panicking. Total: 15 tests, all green. Covers the happy paths, the common failure modes (missing fields, overfull arrays, length violations), and the dispatch-error path (wrong artifact type into wrong validator). Still open from Phase 43 (v2 work, beyond scaffold): - FillValidator catalog integration (worker-existence, status, geo/role match) — needs catalog handle in constructor - EmailValidator PII scan (shared::pii::strip_pii integration) + name-consistency cross-check - Execution loop wiring: generate → validate → observer correction + retry (bounded by max_iterations=3) — spans crates, follow-up - Observer logging: validation results to data/_observer/ops.jsonl and data/_kb/outcomes.jsonl - Scenario fixture tests against tests/multi-agent/playbooks/* Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:35:22 -05:00
root	2f1b9c9768	phase-39+41: land promised artifacts — providers.toml, activation.rs, profiles/ ... Three PRD gaps closed in one coherent batch — all were cosmetic or scaffold-shaped, now real files: Phase 39 (PRD:57): + config/providers.toml — provider registry (name/base_url/auth/ default_model) for ollama, ollama_cloud, openrouter. Commented stubs for gemini + claude pending adapter work. Secrets stay in /etc/lakehouse/secrets.toml or env, NEVER inline. Phase 41 (PRD:115): + crates/vectord/src/activation.rs — ActivationTracker with the PRD-named single-flight guard ("refuse new activation if one is pending/running"). Per-profile granularity — activating A doesn't block B. 5 tests cover the full state machine. Handler body stays in service.rs for now; tracker usage integration is a follow-up. Phase 41 (PRD:113): + crates/shared/src/profiles/ with 4 submodules: * execution.rs — `pub use crate::types::ModelProfile as ExecutionProfile` (backward-compat rename per PRD) * retrieval.rs — top_k, rerank_top_k, freshness cutoff, playbook boost, sensitivity-gate enforcement * memory.rs — playbook boost ceiling, history cap, doc staleness, auto-retire-on-failure * observer.rs — failure cluster size, alert cooldown, ring size, langfuse forwarding All fields `#[serde(default)]` so existing ModelProfile files load unchanged. Still open from the same phases: - Gemini + Claude provider adapters (Phase 40 — 100-200 LOC each) - Full activate_profile handler extraction into activation.rs (Phase 41 — module-structure refactor) - Catalogd CRUD endpoints for retrieval/memory/observer profiles (Phase 41 — exists at list level, no create/update/delete yet) - truth/ repo-root directory for file-backed rules (Phase 42 — TOML loader + schema) - crates/validator crate (Phase 43 — full greenfield) Workspace warnings still at 0. 5 new tests, all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:32:40 -05:00
root	049a4b69fb	truth: split staffing + devops into dedicated modules (Phase 42 PRD) ... Phase 42 PRD (docs/CONTROL_PLANE_PRD.md:137) specified: - crates/truth/src/staffing.rs — staffing rule shapes - crates/truth/src/devops.rs — scaffold for DevOps long-horizon PHASES.md marked Phase 42 done, but the rule sets lived inline in default_truth_store() in lib.rs. Worked, but doesn't match the PRD's module separation — and that separation matters when the long-horizon phase fleshes out devops rules: "Keeps the dispatcher signature stable so no refactor needed later." Fix: extract staffing_rules() into staffing.rs (5 rules, unchanged behavior) + create devops.rs with an empty scaffold. default_truth_store becomes a one-line composition: devops::devops_rules(staffing::staffing_rules(TruthStore::new())) 4 new tests in the submodules cover: - staffing_rules registers expected count (regression guard) - blacklisted worker fails the client-not-blacklisted rule - missing deadline fires Reject via FieldEmpty condition - devops scaffold is a no-op for now Total truth tests: 28 → 32. Workspace warnings still at 0. Still open from Phase 42 (flagged, not in this commit): - `truth/` dir at repo root for file-backed rule loading (TOML/YAML). Rules are in-code today; loader work is a separate feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:25:54 -05:00
root	08cc960115	vectord: Phase 41 gate fixes — 202 ACCEPTED + /profile/jobs/{id} alias ... Phase 41 PRD (docs/CONTROL_PLANE_PRD.md:121) gate: "Activate a profile → returns 202 in <100ms → job completes in background → /vectors/profile/jobs/{id} shows progress" Two concrete mismatches to PRD: 1. activate_profile returned HTTP 200, not 202. Fix: wrap the Json return in (StatusCode::ACCEPTED, Json(...)) so the async semantics are visible at the status-code level. 2. The PRD quotes GET /vectors/profile/jobs/{id} but code only exposed /vectors/jobs/{id}. Fix: add an alias route — same get_job handler, second URL matches what the PRD's polling example documents. Still open from Phase 41 (flagged for follow-up, bigger scope): - crates/shared/src/profiles/ module with ExecutionProfile, RetrievalProfile, MemoryProfile, ObserverProfile types — PRD claims them, file doesn't exist; ModelProfile still does all four roles today. This is a real schema-refactor, not 6-line work. - crates/vectord/src/activation.rs with ActivationTracker — the activation logic lives inline in service.rs; extracting it is a module-structure change. - Phase 37 hot-swap stress test in tests/multi-agent/run_stress.ts Phase 3 — PRD says it must pass, current state unknown. Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:21:49 -05:00
root	999abd6999	gateway/v1: model-prefix routing closes Phase 39 PRD gate ... Phase 39 PRD (docs/CONTROL_PLANE_PRD.md:62) promised: "/v1/chat routes by `model` field: prefix match (e.g. openrouter/anthropic/claude-3.5-sonnet → OpenRouter; bare names → Ollama)" Actual behavior required clients to pass `provider: "openrouter"` explicitly. Bare `model: "openrouter/..."` would fall through to the "unknown provider ''" error. PRD gate never actually passed. Fix: resolve_provider(&ChatRequest) picks (provider, effective_model): - explicit `req.provider` wins, model passes through unchanged - else strip "openrouter/" prefix → provider="openrouter", model without prefix (OpenRouter API expects "openai/gpt-4o-mini", not "openrouter/openai/gpt-4o-mini") - else strip "cloud/" prefix → provider="ollama_cloud" - else default provider="ollama" Adapter calls use Cow<ChatRequest>: borrowed when no strip needed (zero alloc), owned when we needed to build a new model string. Keeps the hot path allocation-free for the common case. ChatRequest gains #[derive(Clone)] — needed for the Owned variant. 5 new tests pin the resolution semantics including the "explicit provider + prefixed model" corner case (trust the caller, don't double-strip). Workspace warnings unchanged at 0. Still not shipped from Phase 39: config/providers.toml — hardcoded match arms work fine in practice, centralizing them is cosmetic. Flag as a follow-up if a 4th provider lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 13:16:36 -05:00
root	81bae108f4	gateway/tools: collapse ToolRegistry::new() and new_with_defaults() into one ... Two constructors existed with a subtle trap: - `new()` had `#[allow(dead_code)]` and called `register_defaults()` via `tokio::task::block_in_place(...)` — a sync wrapper hack around an async method, fragile and unused. - `new_with_defaults()` was misleadingly named — it created the empty registry WITHOUT registering defaults, despite the name. main.rs was doing the right thing: `new_with_defaults()` + explicit `.register_defaults().await`. The misleading name was a landmine for future callers. Fix: delete the dead `new()` with its block_in_place hack, rename `new_with_defaults()` → `new()` (Rust idiom — `new` is the canonical constructor), add a docstring that says what you need to do after. Single clear API. Update the one caller in main.rs. Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:44:18 -05:00
root	5df4d48109	cleanup: drop two #[allow] attributes that were hiding real dead code ... - ingestd/src/service.rs: top-of-file `#[allow(unused_imports)]` was masking genuinely unused `delete` and `patch` routing constructors in an axum import block. Removed the attribute, trimmed the imports to only `get` and `post` (what's actually used). Any future over-import now trips the unused_imports lint immediately instead of being silently allowed. - gateway/src/v1/truth.rs: `truth_router()` was a 4-line stub wrapping a single `/context` route — carried `#[allow(dead_code)]` because v1/mod.rs wires `get(truth::context)` directly onto its own router, bypassing this helper. Zero callers across the workspace. Deleted the function + allow + now-unused Router import. Left a breadcrumb comment pointing to the real wiring. Workspace warnings: 0 (lib + tests). Each #[allow] removed raises the bar on future code entering these modules — the linter now catches the same classes of bugs at PR time. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:42:49 -05:00
root	ffdc842ec3	ingestd: scope test-only imports into the test module ... schema_evolution.rs had two `#[allow(unused_imports)]` attributes hiding over-broad top-level imports: - `Schema` was imported at crate level but only used in test code - `Arc` was imported at crate level but only used in test code - `DataType` and `SchemaRef` were actually used (28 references) — the allow on that line was cargo-culted. Fix: drop the allows, move Schema + Arc into the #[cfg(test)] block where they're actually used. The non-test build no longer imports symbols it doesn't need. Test build still works because the imports are now in the test module's scope. Workspace warnings still at 0 (lib + tests). Net: -3 import lines from crate scope, +2 into test scope. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:41:15 -05:00
root	12e615bb5d	ingestd/vectord: remove two fragile unwraps on Option paths ... Both were technically safe — guarded above by map_or(true, ...) and Some(entry) assignment respectively — but relied on multi-line invariants that a future refactor could easily break. - ingestd/watcher.rs:80: path.file_name().unwrap() on a path that was already checked via map_or(true, ...) two lines up. Fix: let-else binds filename once, no double lookup, no unwrap. - vectord/promotion.rs:145: file.current.as_ref().unwrap() called TWICE on the same line to log config + trial_id. Guard via `if let Some(cur) = &file.current` so the log gracefully skips if the invariant ever breaks instead of panicking at runtime. Both are drop-in semantically: happy path identical, error path now graceful-skip instead of panic. Workspace warnings still at 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:39:40 -05:00
root	a934a76988	aibridge: delete deprecated estimate_tokens wrapper — fully migrated ... cdc24d8 migrated all 5 call sites to shared::model_matrix::ModelMatrix. Grep across the workspace confirms zero remaining callers (only doc comments in the new module reference the old name). Wrapper was there to smooth the transition; transition is done. Leaves a 3-line breadcrumb comment pointing to the new location so anyone opening this file sees the migration history. The deprecated wrapper itself is 4 lines deleted. Workspace warnings still at 0 (both lib + tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:38:01 -05:00
root	cdc24d8bd0	shared: build ModelMatrix — migrate 5 call sites off deprecated estimate_tokens ... The `aibridge::context::estimate_tokens` deprecation has been pointing at `shared::model_matrix::ModelMatrix::estimate_tokens` for a while, but that module didn't exist — so the deprecation was aspirational noise, not actionable guidance. Built the minimal target: `shared::model_matrix::ModelMatrix` with an associated `estimate_tokens(text: &str) -> usize` method. Same chars/4 ceiling heuristic as the deprecated helper. 6 tests cover empty/3/4/5-char cases, multi-byte UTF-8 (emoji count as 1 char each), and linear scaling to 400-char inputs. Migrated 5 call sites: - aibridge/context.rs:88 — opts.system token count - aibridge/context.rs:89 — prompt token count - aibridge/tree_split.rs:22 — import (now uses ModelMatrix) - aibridge/tree_split.rs:84, 89 — truncate_scratchpad budget loop - aibridge/tree_split.rs:282 — scratchpad post-truncation assertion - aibridge/context.rs:183 — system-prompt budget test Also cleaned up two parallel test warnings: - aibridge/context.rs legacy estimate_tokens_ceiling_divides_by_four test deleted (ModelMatrix's tests cover the same behavior now). - vectord/playbook_memory.rs:1650 unused_mut on e_alive. Net workspace warning count: 11 → 0 (including --tests build). The deprecated `estimate_tokens` wrapper stays in aibridge/context.rs for external callers. Future commits can remove it entirely once no public API surface still references it. The applier's warning-count gate now has a floor of 0 — any future patch that introduces a single warning trips the gate automatically. Previously a floor of 11 tolerated noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:32:16 -05:00
root	fdc5123f6d	cleanup: drop workspace warnings from 11 to 6 ... Three trivial cleanups that pull the workspace baseline down by five: - vectord/trial.rs: removed unused ObjectStore import (not referenced anywhere in the file; cargo's unused_imports lint was flagging it on every check). Net: -2 warnings (cascade effect from one import). - ui/main.rs:1241: `Err(e)` with unused binding → `Err(_)`. - ui/main.rs:1247: `let mut import_table` never mutated → `let`. Matters because the scrum_applier's hardened warning-count gate uses this baseline as its reject threshold. Lower baseline = lower floor = any future patch that adds a warning trips the gate earlier. Remaining 6 warnings are all aibridge context::estimate_tokens deprecation notices pointing at a planned-but-unbuilt shared::model_matrix::ModelMatrix::estimate_tokens. Fix requires creating that type (next commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 06:28:36 -05:00