profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: close 2 cross-runtime parity gaps + document unified log

Browse Source 
Companion to lakehouse 98b6647. Architecture comparison decisions
tracker now captures:

  - Go validatord direct header read (fixes 6847bbc): closes the
    case where Langfuse-off middleware passthrough silently dropped
    forwarded X-Lakehouse-Trace-Id
  - Rust IterateResponse trace_id echo (fixes 98b6647): closes the
    asymmetry where Go's response carried the join key and Rust's
    didn't
  - Unified longitudinal log demonstrated end-to-end: both daemons
    co-writing /tmp/lakehouse-validator/sessions.jsonl, distinct
    daemon tags, one DuckDB query covers both

24/24 parity assertions (validator 6/6, extract_json 12/12,
session_log 4/4, materializer 2/2) hold against live :3100 + :4110.
Both runtimes deployed with today's full stack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root 2026-05-02 06:25:21 -05:00
parent 6847bbc180
commit a21a34b057
6 changed files with 6 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
STATE_OF_PLAY.md
View File

@ -1,6 +1,6 @@
# STATE OF PLAY — Lakehouse-Go
**Last verified:** 2026-05-02 ~10:30 CDT
**Last verified:** 2026-05-02 ~11:30 CDT
**Verified by:** **production-readiness gauntlet** — 21/21 smoke chain green, per-component scrum across 4 bundles, **3 cross-runtime parity probes all green post-fix** (validator: **6/6 match** after wire-format alignment shipped; materializer: 2/2 after omitempty fix; extract_json: 12/12). All findings surfaced by the parity probes have been actioned. Disposition: `reports/cutover/gauntlet_2026-05-02/disposition.md`.
> **Read this FIRST.** When the user says "we're working on lakehouse," default to the Go rewrite (this repo); the Rust legacy at `/home/profit/lakehouse/` is maintenance-only. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.

1
docs/ARCHITECTURE_COMPARISON.md
View File

@ -50,6 +50,7 @@ Don't:
| 2026-05-02 | **Substrate fix verified at original failure scale** | Re-ran multitier 5min @ conc=50 (the footprint that originally surfaced the bug at 96-98% fail). Result: 132,211 scenarios at 438/sec, **6/6 classes at 0% failure**. Throughput dropped 1,115/sec → 438/sec because broken scenarios now do real HNSW Add work. Tails healthy: surge_fill_validate p99=1.53s, playbook_record_replay p99=2.32s. **Fix scales — closing the open thread.** |
| 2026-05-02 | **Drop Python sidecar from Rust aibridge — DONE** | `crates/aibridge/src/client.rs` rewrite (commit `ba928b1` in lakehouse). AiClient now talks Ollama directly: per-text `/api/embed` loop, `/api/generate` for chat + rerank-loop + admin (unload/preload), `/api/ps` + `std::process::Command nvidia-smi` for vram_snapshot. Public API unchanged — 0 callers updated. Verification: cargo test -p aibridge 32/32 PASS, live smoke `/ai/embed` returns 768-dim vector + `/v1/chat` returns "OK". Sidecar's `lab_ui.py` + `pipeline_lab.py` (~888 LOC dev-only UIs) keep running; only the hot-path embed/generate/rerank/admin routes are retired. Process count drops from "mega-binary + sidecar" to "mega-binary alone". |
| 2026-05-02 | **Rust observability parity (trace-id + session JSONL) — DONE** | Mirrors the Go-side wave (commits `d6d2fdf` + `1a3a82a` in golangLAKEHOUSE). `crates/gateway/src/v1/iterate.rs` now reads `X-Lakehouse-Trace-Id` header, forwards to `/v1/chat` + `/v1/validate` hops, emits per-attempt Langfuse spans (`iterate.attempt[N]`), and writes one `SessionRecord` JSONL row per session via the new `crates/gateway/src/v1/session_log.rs` writer. New `crates/gateway/src/bin/parity_session_log` helper enables the cross-runtime parity probe at `scripts/cutover/parity/session_log_parity.sh` — **4/4 fixtures byte-equal** (including unicode prompts) after normalizing the 2 fields that legitimately differ (timestamp, daemon name). Both runtimes can write to the same path and DuckDB queries see them as one stream. 90/90 Rust unit tests PASS, 33 Go packages PASS. |
| 2026-05-02 | **Cross-runtime parity gap closure — DONE** | Two follow-ups from the deploy + reality wave: (1) Go validatord didn't honor `X-Lakehouse-Trace-Id` when Langfuse middleware was a passthrough (no env config) — fixed by reading the header DIRECTLY in the iterate handler before the ctx fallback (Go commit `6847bbc`). (2) Rust `IterateResponse`/`IterateFailure` didn't echo `trace_id` back to callers — added the field with skip-serializing-if-none for backward compat (Rust commit `98b6647`). Plus enabled `[gateway].session_log_path` on live Rust pointing at the SAME path Go writes to, so DuckDB queries see one unified longitudinal log. **Verified end-to-end**: identical /v1/iterate to both runtimes lands in one sessions.jsonl tagged `daemon=gateway` (Rust) vs `daemon=validatord` (Go); response bodies both echo the forwarded trace_id. 24/24 parity assertions (validator+extract_json+session_log+materializer) hold post-restart. |
| 2026-05-02 | **Port Rust materializer to Go (transforms.ts) — DONE** | `internal/materializer` + `cmd/materializer` + `materializer_smoke.sh`. Ports `transforms.ts` (12 transforms) + `build_evidence_index.ts`. Idempotency, day-partition, receipt. 14 tests green; on-wire JSON matches TS so both runtimes interoperate. |
| 2026-05-02 | **Port Rust replay tool to Go — DONE** | `internal/replay` + `cmd/replay` + `replay_smoke.sh`. Ports `replay.ts` retrieve → bundle → /v1/chat → validate → log. Closes audit-FULL phase 7 live invocation on Go side. 14 tests green; same `data/_kb/replay_runs.jsonl` shape (schema=replay_run.v1) as TS. |
| 2026-05-02 | **`/v1/validate` + `/v1/iterate` HTTP surface — DONE** | `cmd/validatord` (port 3221) hosts both endpoints. `internal/validator` gains `PlaybookValidator` (3rd kind), JSONL roster loader, and the `Iterate` orchestrator + `ExtractJSON` helper. Gateway proxies `/v1/validate` + `/v1/iterate` to validatord. Closes the last "Go-primary" backlog item (architecture_comparison.md item #7). 30+ tests + `validatord_smoke.sh` 5/5 PASS. |

2
reports/cutover/gauntlet_2026-05-02/parity/extract_json_parity.md
View File

@ -1,6 +1,6 @@
# extract_json parity probe — Rust vs Go
**Date:** 2026-05-02T09:42:37Z
**Date:** 2026-05-02T11:24:21Z
**Rust helper:** `/home/profit/lakehouse/target/release/parity_extract_json` (links live `gateway::v1::iterate::extract_json`)
**Go helper:** `./bin/parity_extract_json_go` (links live `internal/validator.ExtractJSON`)

2
reports/cutover/gauntlet_2026-05-02/parity/materializer_parity.md
View File

@ -1,6 +1,6 @@
# Materializer parity probe — Bun vs Go
**Date:** 2026-05-02T09:38:32Z
**Date:** 2026-05-02T11:24:21Z
**Bun:** `/home/profit/lakehouse/scripts/distillation/build_evidence_index.ts`
**Go:** `./bin/materializer`

2
reports/cutover/gauntlet_2026-05-02/parity/session_log_parity.md
View File

@ -1,6 +1,6 @@
# session_log parity probe — Rust gateway vs Go validatord
**Date:** 2026-05-02T10:53:00Z
**Date:** 2026-05-02T11:24:21Z
**Rust helper:** `/home/profit/lakehouse/target/release/parity_session_log`
**Go helper:** `./bin/parity_session_log_go`

2
reports/cutover/gauntlet_2026-05-02/parity/validator_parity.md
View File

@ -1,6 +1,6 @@
# Validator parity probe — Rust :3100 vs Go :4110
**Date:** 2026-05-02T10:52:59Z
**Date:** 2026-05-02T11:24:15Z
**Rust gateway:** `http://127.0.0.1:3100` · **Go gateway:** `http://127.0.0.1:4110`
Identical `POST /v1/validate` request → both runtimes. Match