Surfaced during the 2026-05-02 deploy + reality wave: the persistent
Go stack runs without LANGFUSE_URL/PUBLIC_KEY/SECRET_KEY env, so
shared.langfuseMiddleware operates as a passthrough — never minting
a trace id, never stashing it on the request context. Result:
session_id was empty on every JSONL row, breaking correlation across
the longitudinal log + replay_runs.jsonl + future Langfuse traces.
The fix: validatord falls back to a locally-generated time-ordered
hex id when both the X-Lakehouse-Trace-Id header AND the middleware
context are empty. Same shape Langfuse accepts, so a future deploy
that turns Langfuse on doesn't break correlation — already-emitted
session_ids stay valid as Langfuse trace ids.
Verified post-deploy by driving 9 /v1/iterate sessions through the
persistent stack at :4110:
- 6 accepted on iter 0 (qwen2.5:latest first-shot 75%)
- 2 max_iter_exhausted (no_json on prose-y prompts)
- 1 infra_error (chatd cold-start probe timed out at 5s)
Latest row's session_id: "18abbabdc2306a83-c2306aa9" (was: "")
Probe re-runs (validator_parity, session_log_parity) included as
post-deploy artifacts; both 6/6 + 4/4 with the freshly-restarted
persistent gateway+validatord binaries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to lakehouse commit 57bde63 (Rust gateway gains
trace-id propagation + coordinator session JSONL). The
cross-runtime parity probe is the regression gate that prevents
silent schema drift between the two runtimes.
scripts/cutover/parity/session_log_parity.sh:
- 4 fixtures (accepted_grounded, max_iter_exhausted, infra_error,
unicode_in_prompt) feed identical input to both helpers
- jq -e validity gate + non-trivial-equal guard prevents the
"both sides fail identically → spurious match" failure mode
(caught one IFS='||' bug during initial authoring — recorded
in the script comment)
- normalize() strips timestamp + daemon (legitimate per-producer
differences); everything else must be byte-equal
- Result: 4/4 fixtures match, including unicode
scripts/cutover/parity/session_log_helper/main.go:
- Tiny stdin/stdout Go helper that round-trips a fixture
through validator.SessionRecord serde
- Counterpart to crates/gateway/src/bin/parity_session_log.rs
docs/ARCHITECTURE_COMPARISON.md decisions tracker:
- "Rust observability parity" row added (DONE 2026-05-02)
- Cross-runtime probe documented as reusable gate
STATE_OF_PLAY refreshed.
Both observability pieces (trace-id propagation, session JSONL)
now exist on both runtimes. Operators who point Rust gateway and
Go validatord at the same session-log path get a unified
longitudinal stream queryable via DuckDB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
