98b6647f2a
Some checks failed
lakehouse/auditor 14 blocking issues: cloud: claim not backed — "Verified end-to-end against persistent Go stack on :4110:"
Closes the 2026-05-02 cross-runtime parity gap: Go's
validator.IterateResponse carried trace_id back to callers; Rust's
didn't. A caller pivoting from response → Langfuse → session log
worked on Go but failed on Rust because the join key wasn't visible
in the response body.
## Changes
crates/gateway/src/v1/iterate.rs:
- IterateResponse + IterateFailure gain `trace_id: Option<String>`
(skip-serializing-if-none preserves backward-compat for any
consumer parsing the response without the field)
- Both return sites populated with the resolved trace_id
lakehouse.toml:
- [gateway].session_log_path set to /tmp/lakehouse-validator/sessions.jsonl
— same path Go validatord writes to. The two daemons now co-write
one unified longitudinal log; rows tag daemon="gateway" vs
daemon="validatord" so producers stay distinguishable in DuckDB
queries. Append-write is atomic at the row sizes both runtimes
produce, so concurrent writes from both daemons are safe.
## Verification
Post-restart of lakehouse.service:
POST /v1/iterate with X-Lakehouse-Trace-Id: rust-fix1-test
→ response.trace_id = "rust-fix1-test" ✓ (was: field absent)
→ sessions.jsonl latest row daemon=gateway, session_id=rust-fix1-test ✓ (was: no row)
Cross-runtime drive — same prompt to Rust :3100 and Go :4110:
Rust: trace_id=unified-rust-001, daemon=gateway, accepted
Go: trace_id=unified-go-001, daemon=validatord, accepted
Same file, distinct daemons, one query covers both:
SELECT daemon, COUNT(*) FROM read_json_auto('sessions.jsonl', format='nd') GROUP BY daemon
→ gateway: 2, validatord: 19
All 4 parity probes still 6/6 + 12/12 + 4/4 + 2/2 against live
:3100 + :4110 stacks. Cargo test 4/4 PASS for v1::iterate module.
## Architecture invariant
The "unified longitudinal log" thesis is now demonstrated. Operators
running both runtimes in production point both daemons at the same
session_log_path and DuckDB queries naturally span both producers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
100 lines
3.3 KiB
TOML
100 lines
3.3 KiB
TOML
# Lakehouse Configuration
|
|
|
|
[gateway]
|
|
host = "0.0.0.0"
|
|
port = 3100
|
|
# Coordinator session JSONL — one row per /v1/iterate session for
|
|
# offline DuckDB analysis. Cross-runtime parity with the Go-side
|
|
# [validatord].session_log_path. Set to the SAME path Go validatord
|
|
# writes to so DuckDB queries see one unified longitudinal stream
|
|
# across both runtimes (rows are tagged daemon="gateway" vs
|
|
# daemon="validatord" so producers stay distinguishable). Append-write
|
|
# is atomic at the row sizes both runtimes produce — both daemons
|
|
# co-writing is safe.
|
|
session_log_path = "/tmp/lakehouse-validator/sessions.jsonl"
|
|
|
|
[storage]
|
|
root = "./data"
|
|
profile_root = "./data/_profiles"
|
|
rescue_bucket = "rescue"
|
|
|
|
[[storage.buckets]]
|
|
name = "primary"
|
|
backend = "local"
|
|
root = "./data"
|
|
|
|
[[storage.buckets]]
|
|
name = "rescue"
|
|
backend = "local"
|
|
root = "./data/_rescue"
|
|
|
|
[[storage.buckets]]
|
|
name = "testing"
|
|
backend = "local"
|
|
root = "./data/_testing"
|
|
|
|
# S3 bucket via MinIO. The name "s3:lakehouse" is the convention
|
|
# lance_backend.rs uses to emit s3:// URIs for Lance datasets.
|
|
# Credentials resolved via environment (AWS_ACCESS_KEY_ID etc) or
|
|
# the secrets provider.
|
|
[[storage.buckets]]
|
|
name = "s3:lakehouse"
|
|
backend = "s3"
|
|
bucket = "lakehouse"
|
|
endpoint = "http://localhost:9000"
|
|
region = "us-east-1"
|
|
secret_ref = "minio-lakehouse"
|
|
|
|
[catalog]
|
|
# Manifests persisted to object storage under this prefix
|
|
manifest_prefix = "_catalog/manifests"
|
|
|
|
[query]
|
|
# max_rows_per_query = 10000
|
|
|
|
[sidecar]
|
|
# Post-2026-05-02: AiClient talks directly to Ollama; the Python
|
|
# sidecar's hot-path role (~120 LOC of pure Ollama wrappers) was
|
|
# retired. Field name kept for migration compat — value now points
|
|
# at Ollama on :11434. Lab UI + pipeline_lab Python remains as a
|
|
# dev-only tool, NOT on this URL.
|
|
url = "http://localhost:11434"
|
|
|
|
[ai]
|
|
embed_model = "nomic-embed-text"
|
|
# Local-tier defaults bumped 2026-04-30: qwen3.5:latest is the
|
|
# stronger local rung in the 5-loop substrate (per
|
|
# project_small_model_pipeline_vision.md). Same JSON-clean property
|
|
# as qwen2.5, more capacity. Ollama still serves both — bump back
|
|
# in this file if a workload regressed.
|
|
gen_model = "qwen3.5:latest"
|
|
rerank_model = "qwen3.5:latest"
|
|
|
|
[auth]
|
|
enabled = false
|
|
# api_key = "changeme"
|
|
|
|
[observability]
|
|
# Export traces to stdout (set to "otlp" for OpenTelemetry collector)
|
|
exporter = "stdout"
|
|
service_name = "lakehouse"
|
|
|
|
[agent]
|
|
# Phase 16.2 — background autotune agent. Opt-in: set enabled = true to
|
|
# let the agent continuously propose + trial HNSW configs and auto-promote
|
|
# winners. Defaults are conservative so it stays out of the way of live
|
|
# search traffic on shared Ollama.
|
|
enabled = true
|
|
cycle_interval_secs = 120 # periodic wake if no triggers
|
|
cooldown_between_trials_secs = 10 # min gap between trials
|
|
min_recall = 0.9 # never promote below this
|
|
max_trials_per_hour = 20 # hard budget cap
|
|
|
|
# Model roster — available for profile hot-swap
|
|
# qwen3.5:latest: stronger local rung — JSON-clean, 8K+ context,
|
|
# default for gen_model and rerank_model
|
|
# qwen3: 8.2B, 40K context, thinking+tools, best for reasoning tasks
|
|
# qwen2.5: 7B, 8K context, fast — kept loaded for the 2026-04 era
|
|
# comparison runs; new defaults use qwen3.5:latest
|
|
# nomic-embed-text: 137M, embedding-only, used by all profiles
|