diff --git a/STATE_OF_PLAY.md b/STATE_OF_PLAY.md index 0d9c5b8..fc862da 100644 --- a/STATE_OF_PLAY.md +++ b/STATE_OF_PLAY.md @@ -1,6 +1,6 @@ # STATE OF PLAY — Lakehouse-Go -**Last verified:** 2026-05-02 ~10:30 CDT +**Last verified:** 2026-05-02 ~11:30 CDT **Verified by:** **production-readiness gauntlet** — 21/21 smoke chain green, per-component scrum across 4 bundles, **3 cross-runtime parity probes all green post-fix** (validator: **6/6 match** after wire-format alignment shipped; materializer: 2/2 after omitempty fix; extract_json: 12/12). All findings surfaced by the parity probes have been actioned. Disposition: `reports/cutover/gauntlet_2026-05-02/disposition.md`. > **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo); the Rust legacy at `/home/profit/lakehouse/` is maintenance-only. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes. diff --git a/docs/ARCHITECTURE_COMPARISON.md b/docs/ARCHITECTURE_COMPARISON.md index 65bfcce..c9e3d44 100644 --- a/docs/ARCHITECTURE_COMPARISON.md +++ b/docs/ARCHITECTURE_COMPARISON.md @@ -50,6 +50,7 @@ Don't: | 2026-05-02 | **Substrate fix verified at original failure scale** | Re-ran multitier 5min @ conc=50 (the footprint that originally surfaced the bug at 96-98% fail). Result: 132,211 scenarios at 438/sec, **6/6 classes at 0% failure**. Throughput dropped 1,115/sec → 438/sec because broken scenarios now do real HNSW Add work. Tails healthy: surge_fill_validate p99=1.53s, playbook_record_replay p99=2.32s. **Fix scales — closing the open thread.** | | 2026-05-02 | **Drop Python sidecar from Rust aibridge — DONE** | `crates/aibridge/src/client.rs` rewrite (commit `ba928b1` in lakehouse). AiClient now talks Ollama directly: per-text `/api/embed` loop, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `std::process::Command nvidia-smi` for vram_snapshot. Public API unchanged — 0 callers updated. Verification: cargo test -p aibridge 32/32 PASS, live smoke `/ai/embed` returns 768-dim vector + `/v1/chat` returns "OK". Sidecar's `lab_ui.py` + `pipeline_lab.py` (~888 LOC dev-only UIs) keep running; only the hot-path embed/generate/rerank/admin routes are retired. Process count drops from "mega-binary + sidecar" to "mega-binary alone". | | 2026-05-02 | **Rust observability parity (trace-id + session JSONL) — DONE** | Mirrors the Go-side wave (commits `d6d2fdf` + `1a3a82a` in golangLAKEHOUSE). `crates/gateway/src/v1/iterate.rs` now reads `X-Lakehouse-Trace-Id` header, forwards to `/v1/chat` + `/v1/validate` hops, emits per-attempt Langfuse spans (`iterate.attempt[N]`), and writes one `SessionRecord` JSONL row per session via the new `crates/gateway/src/v1/session_log.rs` writer. New `crates/gateway/src/bin/parity_session_log` helper enables the cross-runtime parity probe at `scripts/cutover/parity/session_log_parity.sh` — **4/4 fixtures byte-equal** (including unicode prompts) after normalizing the 2 fields that legitimately differ (timestamp, daemon name). Both runtimes can write to the same path and DuckDB queries see them as one stream. 90/90 Rust unit tests PASS, 33 Go packages PASS. | +| 2026-05-02 | **Cross-runtime parity gap closure — DONE** | Two follow-ups from the deploy + reality wave: (1) Go validatord didn't honor `X-Lakehouse-Trace-Id` when Langfuse middleware was a passthrough (no env config) — fixed by reading the header DIRECTLY in the iterate handler before the ctx fallback (Go commit `6847bbc`). (2) Rust `IterateResponse`/`IterateFailure` didn't echo `trace_id` back to callers — added the field with skip-serializing-if-none for backward compat (Rust commit `98b6647`). Plus enabled `[gateway].session_log_path` on live Rust pointing at the SAME path Go writes to, so DuckDB queries see one unified longitudinal log. **Verified end-to-end**: identical /v1/iterate to both runtimes lands in one sessions.jsonl tagged `daemon=gateway` (Rust) vs `daemon=validatord` (Go); response bodies both echo the forwarded trace_id. 24/24 parity assertions (validator+extract_json+session_log+materializer) hold post-restart. | | 2026-05-02 | **Port Rust materializer to Go (transforms.ts) — DONE** | `internal/materializer` + `cmd/materializer` + `materializer_smoke.sh`. Ports `transforms.ts` (12 transforms) + `build_evidence_index.ts`. Idempotency, day-partition, receipt. 14 tests green; on-wire JSON matches TS so both runtimes interoperate. | | 2026-05-02 | **Port Rust replay tool to Go — DONE** | `internal/replay` + `cmd/replay` + `replay_smoke.sh`. Ports `replay.ts` retrieve → bundle → /v1/chat → validate → log. Closes audit-FULL phase 7 live invocation on Go side. 14 tests green; same `data/_kb/replay_runs.jsonl` shape (schema=replay_run.v1) as TS. | | 2026-05-02 | **`/v1/validate` + `/v1/iterate` HTTP surface — DONE** | `cmd/validatord` (port 3221) hosts both endpoints. `internal/validator` gains `PlaybookValidator` (3rd kind), JSONL roster loader, and the `Iterate` orchestrator + `ExtractJSON` helper. Gateway proxies `/v1/validate` + `/v1/iterate` to validatord. Closes the last "Go-primary" backlog item (architecture_comparison.md item #7). 30+ tests + `validatord_smoke.sh` 5/5 PASS. | diff --git a/reports/cutover/gauntlet_2026-05-02/parity/extract_json_parity.md b/reports/cutover/gauntlet_2026-05-02/parity/extract_json_parity.md index aba21d7..6db9ffd 100644 --- a/reports/cutover/gauntlet_2026-05-02/parity/extract_json_parity.md +++ b/reports/cutover/gauntlet_2026-05-02/parity/extract_json_parity.md @@ -1,6 +1,6 @@ # extract_json parity probe — Rust vs Go -**Date:** 2026-05-02T09:42:37Z +**Date:** 2026-05-02T11:24:21Z **Rust helper:** `/home/profit/lakehouse/target/release/parity_extract_json` (links live `gateway::v1::iterate::extract_json`) **Go helper:** `./bin/parity_extract_json_go` (links live `internal/validator.ExtractJSON`) diff --git a/reports/cutover/gauntlet_2026-05-02/parity/materializer_parity.md b/reports/cutover/gauntlet_2026-05-02/parity/materializer_parity.md index 39bd92d..eb0d62a 100644 --- a/reports/cutover/gauntlet_2026-05-02/parity/materializer_parity.md +++ b/reports/cutover/gauntlet_2026-05-02/parity/materializer_parity.md @@ -1,6 +1,6 @@ # Materializer parity probe — Bun vs Go -**Date:** 2026-05-02T09:38:32Z +**Date:** 2026-05-02T11:24:21Z **Bun:** `/home/profit/lakehouse/scripts/distillation/build_evidence_index.ts` **Go:** `./bin/materializer` diff --git a/reports/cutover/gauntlet_2026-05-02/parity/session_log_parity.md b/reports/cutover/gauntlet_2026-05-02/parity/session_log_parity.md index b57205a..97e8f07 100644 --- a/reports/cutover/gauntlet_2026-05-02/parity/session_log_parity.md +++ b/reports/cutover/gauntlet_2026-05-02/parity/session_log_parity.md @@ -1,6 +1,6 @@ # session_log parity probe — Rust gateway vs Go validatord -**Date:** 2026-05-02T10:53:00Z +**Date:** 2026-05-02T11:24:21Z **Rust helper:** `/home/profit/lakehouse/target/release/parity_session_log` **Go helper:** `./bin/parity_session_log_go` diff --git a/reports/cutover/gauntlet_2026-05-02/parity/validator_parity.md b/reports/cutover/gauntlet_2026-05-02/parity/validator_parity.md index ac680dc..19b8b5f 100644 --- a/reports/cutover/gauntlet_2026-05-02/parity/validator_parity.md +++ b/reports/cutover/gauntlet_2026-05-02/parity/validator_parity.md @@ -1,6 +1,6 @@ # Validator parity probe — Rust :3100 vs Go :4110 -**Date:** 2026-05-02T10:52:59Z +**Date:** 2026-05-02T11:24:15Z **Rust gateway:** `http://127.0.0.1:3100` · **Go gateway:** `http://127.0.0.1:4110` Identical `POST /v1/validate` request → both runtimes. Match