golangLAKEHOUSE/docs/SPEC.md

SPEC: Lakehouse-Go Component Port Plan

Status: DRAFT — companion to PRD.md. Component-by-component port plan with library choices, effort estimates, and acceptance gates. Created: 2026-04-28 Owner: J

This spec answers: for each piece of the Rust Lakehouse, what Go library carries it, what the effort looks like, and what gate proves the port is real.

Effort scale (one engineer-week = ~40h focused work):

• S — 1–3 days
• M — 1 engineer-week
• L — 2–3 engineer-weeks
• XL — 1+ months
• HARD — open research, see PRD §Hard problems

---

§1. Component port table — Rust crates

Crate	Rust deps that mattered	Go target	Library	Effort	Risk
gateway	axum, tokio, tonic, tower	cmd/gateway	chi + stdlib net/http + google.golang.org/grpc	L	low — Go's strongest domain
catalogd	parquet-rs, arrow, sqlite	cmd/catalogd	apache/arrow-go/v18, mattn/go-sqlite3	L	low
storaged	object_store, aws-sdk	cmd/storaged	aws-sdk-go-v2, minio-go for MinIO-specific paths	M	low
queryd	datafusion, arrow	cmd/queryd	duckdb/duckdb-go/v2 (cgo, official)	HARD	high — see §3
ingestd	csv, json, lopdf, postgres	cmd/ingestd	stdlib encoding/csv, encoding/json, pdfcpu/pdfcpu, jackc/pgx/v5	L	low
vectord	hora, arrow, hnsw	cmd/vectord	coder/hnsw, apache/arrow-go/v18	L	medium — re-validate HNSW recall
matrix indexer (emergent in Rust — mode.rs + build_*_corpus.ts + observer /relevance)	scripts/build_*_corpus.ts, crates/gateway/src/v1/mode.rs, mcp-server/observer.ts	internal/matrix/ + gateway routes (/v1/matrix/*)	stdlib + vectord client	L	medium — see §3.4. Corpus-as-shard composer; relevance filter; strong-model downgrade gate; multi-corpus retrieve+merge. The learning-loop layer that lifts vectord from "static index" to "meta-index that learns from playbooks."
vectord-lance	lance	DROPPED	n/a	n/a	n/a — Parquet+HNSW only
journald	parquet, arrow	cmd/journald	apache/arrow-go/v18	M	low
aibridge	reqwest	library	net/http + connection pool · anthropics/anthropic-sdk-go available for direct Claude calls (currently routed via opencode)	S	low
chatd (Phase 4 — multi-provider LLM dispatcher)	crates/gateway/src/v1/{ollama_cloud,openrouter,opencode}.rs	cmd/chatd + internal/chat/	stdlib net/http only	DONE	medium — see §3.9. 5-provider routing (ollama / ollama_cloud / openrouter / opencode / kimi) by model-name prefix or :cloud suffix. Replaces the Rust gateway's v1:: adapters.
validator	parquet, custom	library	apache/arrow-go/v18 parquet reader	M	low — port the 24 unit tests as gates
truth	tomli, custom DSL	library	pelletier/go-toml/v2	M	low
proto	tonic-build	proto/ + protoc-gen-go	buf + protoc-gen-go-grpc	S	low
shared	serde, anyhow	library	stdlib encoding/json, errors	S	low
ui	dioxus, wasm	REPLACED	html/template + HTMX	L	medium — see §3
lance-bench	criterion	n/a — dropped with Lance	n/a	n/a	n/a

Total Rust crate port effort: ~12–18 engineer-weeks (3–4 months for one engineer; 6–8 weeks for two).

---

§2. Component port table — TypeScript surfaces

TS surface	Current location	Go target	Library	Effort	Risk
mcp-server/index.ts	Bun, :3700	cmd/mcp	modelcontextprotocol/go-sdk (official Go SDK, v1.5.0, Google-collab)	L	medium — MCP semantics
mcp-server/observer.ts	Bun, :3800	cmd/observer	stdlib net/http, slog	M	low
mcp-server/tracing.ts	Bun, Langfuse client	library	go.opentelemetry.io/otel + Langfuse Go client (or hand-roll)	M	low — Langfuse Go OSS support varies
auditor/*.ts	TS, runs as systemd	cmd/auditor	stdlib + gitea API client	L	medium — auditor cross-lineage logic is intricate
tests/real-world/scrum_master_pipeline.ts	TS, ad-hoc	cmd/scrum	stdlib	L	medium — chunking + embed + ladder logic
tests/real-world/scrum_applier.ts	TS, ad-hoc	cmd/scrum-apply	stdlib + git CLI shell-out	M	medium
bot/propose.ts	TS	cmd/bot	stdlib	S	low
Search demo HTML/JS	static	static (no port)	n/a	n/a	n/a — copied as-is

Total TS port effort: ~6–10 engineer-weeks.

---

§3. Hard problem details

§3.1 — Query engine (DuckDB via cgo)

Library: github.com/duckdb/duckdb-go/v2 — official Go bindings via cgo. (Replaces the legacy marcboeker/go-duckdb, which was deprecated when the DuckDB team and Marc Boeker jointly relocated maintenance to the DuckDB org at v2.5.0. Migration is a one-line gofmt -r rewrite of import paths.) Current version v2.10502.0 (April 2026), DuckDB v1.5.2 compat. Statically links default extensions: ICU, JSON, Parquet, Autocomplete.

API shape (replaces the DataFusion SessionContext pattern):

db, _ := sql.Open("duckdb", "")
defer db.Close()
db.Exec("CREATE VIEW workers AS SELECT * FROM read_parquet('s3://bucket/workers/*.parquet')")
rows, _ := db.Query("SELECT role, count(*) FROM workers WHERE state='IL' GROUP BY role")

Acceptance gates:

• G3.1.A — SELECT * FROM read_parquet('workers_500k.parquet') LIMIT 1 returns a row with the expected schema. Establishes Parquet read works.
• G3.1.B — Hybrid SQL+vector query (the POST /vectors/hybrid surface) returns same workers as the Rust path on the same input, ranked the same way modulo embedding precision.
• G3.1.C — Hot-cache merge-on-read: register a base table + a delta Parquet, query, observe both rows merged with the delta winning on conflict.

Fallback if cgo is rejected: run DuckDB as an external process (duckdb -json -c '...' shelled or HTTP via a thin Go wrapper). Adds operational surface; preserves SQL model.

§3.2 — HNSW index

Library: coder/hnsw — pure-Go HNSW, in-process. Supports add / delete / search / persist.

Open question: does coder/hnsw match the recall@10 we measured on the Rust hora path? Need a calibration test:

• Rebuild lakehouse_arch_v1 (the 1086-chunk arch corpus) in Go.
• Compare recall@10 on a fixed query set to the Rust baseline.
• Acceptance: ≤2% drop or we switch library / parameters.

Persistence format: TBD — coder/hnsw has its own snapshot format; ADR equivalent of ADR-008 (Parquet for embeddings + sidecar HNSW file) needs revisiting in Go to confirm the sidecar format we ship.

Acceptance gates:

• G3.2.A — Build HNSW from a Parquet of 100K vectors in <60s
• G3.2.B — Search 100K vectors at k=10 in <50ms p50
• G3.2.C — Recall@10 within 2% of Rust baseline on lakehouse_arch_v1

§3.4 — Matrix indexer (corpus-as-shard composer)

What it is. The matrix indexer is the layer above vectord that turns a fleet of single-corpus HNSW indexes into a learning meta-index. In the Rust system this is emergent — split between corpus builders (scripts/build_*_corpus.ts), the mode runner (crates/gateway/src/v1/mode.rs), the observer relevance endpoint (mcp-server/observer.ts), and the strong-model downgrade gate (mode.rs::execute). In Go we name it explicitly so future sessions don't reduce it to "vectord."

Why corpus-as-shard, not shard-by-id. Sharding a single index by hash(id) is a pure throughput hack with a recall tax. Sharding by corpus is the existing retrieval shape — lakehouse_arch_v1, lakehouse_symbols_v1, scrum_findings_v1, lakehouse_answers_v1, kb_team_runs_v1, successful_playbooks_live, etc. — each with distinct topology and a distinct retrieval intent. Concurrent Adds parallelize naturally because they go to different corpora; the matrix layer's job is to retrieve+merge across them, filter for relevance, and downgrade composition when strong models prove the matrix is anti-additive.

Components to port (in dependency order):

1. Corpus builders — Go equivalents of scripts/build_*_corpus.ts. For each named corpus, a builder that reads source, splits into chunks per the corpus's schema, embeds via /v1/embed, and adds to a vectord index of the same name. Effort: M for the first builder, S for each subsequent.

2. Multi-corpus retrieve+merge (internal/matrix/retrieve.go) — given a query and a list of corpus names, search each at top_k=K, merge by score, return top N globally. Match Rust's pattern: top_k=6 per corpus, top 8 globally before relevance filter.

3. Relevance filter (internal/matrix/relevance.go) — port the threshold-based filter from mcp-server/observer.ts:/relevance. Drops adjacency-pollution chunks that share a corpus with the hit but aren't actually about the query. LH_RELEVANCE_FILTER / LH_RELEVANCE_THRESHOLD env knobs preserved.

4. Strong-model downgrade gate (internal/matrix/downgrade.go) — port is_weak_model + the codereview_lakehouse → codereview_isolation flip from mode.rs::execute. Pass5 proved composed corpora lose 5/5 vs isolation on grok-4.1-fast (p=0.031); the gate is load-bearing for paid-model retrieval quality.

5. Learning-loop integration — write outcomes back to a playbook-memory corpus (probably lakehouse_answers_v1 analogue). This is what makes the matrix INDEX a learning system rather than static retrieval. Per feedback_meta_index_vision.md: this is the north star, not the data structure.

Gateway routes: /v1/matrix/search (multi-corpus retrieve+merge), /v1/matrix/corpora (list + metadata), /v1/matrix/relevance (filter endpoint, used by both internal callers and external tooling).

Acceptance gates:

• G3.4.A — /v1/matrix/search against ≥3 corpora returns merged top-N with corpus attribution per result.
• G3.4.B — Relevance filter drops at least the threshold-margin chunks on a known adjacency-pollution test case.
• G3.4.C — Strong-model downgrade gate flips composed→isolation when the model is non-weak; bypassed when caller sets force_mode.
• G3.4.D — Concurrent Adds across N=4 corpora parallelize (no shared write-lock); Add throughput scales near-linearly with corpus count.

Persistence: each corpus's vectord index persists via the existing G1P LHV1 format. The matrix layer is stateless above that — corpus list lives in catalog, retrieval params in config.

Why this is its own §3.x: in Rust the matrix indexer was emergent and got reduced to "we have vectord" in earlier port-planning. The SPEC names it explicitly so the port preserves the multi-corpus retrieval shape AND the learning loop, not just the HNSW substrate.

§3.5 — Drift quantification (loop 5 of the PRD)

What it is. PRD names "drift" as the 5th loop: quantify when historical decisions stop matching current reality. Distinct from the rating+distillation loop because drift is MEASUREMENT, not LEARNING. The learning loop says "this match worked, remember it"; the drift loop says "this 4-month-old playbook entry — does it still match what the substrate would surface today?"

What's shipped (commit be65f85):

• SCORER drift: re-runs current distillation.ScoreRecord over historical (EvidenceRecord, persisted_category) pairs and reports mismatches + a sorted shift matrix
• internal/drift/drift.go — pure-function ComputeScorerDrift
• 6 unit tests covering no-drift, shift detection, multi-shift sorted-by-count, includeEntries flag, empty input, scorer-version stamping

Future drift shapes (not shipped):

• PLAYBOOK drift: re-run playbook queries through current matrix-search; recorded answer not in top-K = drift
• EMBEDDING drift: KS-test on vector distribution at T1 vs T2
• AUDIT BASELINE drift: matches Rust audit_baselines.jsonl longitudinal signal

Acceptance gates:

• G3.5.A — A scorer-version bump triggers a non-zero Drifted count on a corpus of historical ScoredRuns where the new logic produces different categories than the persisted ones.
• G3.5.B — ScorerDriftReport.ShiftMatrix is deterministic-ordered (count desc, ties broken alphabetically) so JSON output is stable across runs.

§3.6 — Staffing-side structured filter

What it is. Reality tests on the candidates + workers corpora (commits 0d1553c, a97881d) surfaced that pure semantic retrieval can't gate by location/status/availability — the matrix indexer returns Production Workers for a Forklift+OSHA-30 query because nomic-embed-text's geometry doesn't separate the role labels well. Structured filtering is the addressable piece: pre-filter the candidate set on metadata fields BEFORE semantic ranking.

What's shipped (commit b199093):

• SearchRequest.MetadataFilter — map[string]any of metadata field → expected value (single value or list-of-values for OR semantics within a key, AND across keys)
• Post-retrieval filter applied before top-K truncation in internal/matrix/retrieve.go
• SearchResponse.MetadataFilterDropped for telemetry on filter aggressiveness
• 7 unit tests covering nil filter, missing metadata, exact match, AND across keys, OR within list, bool match, malformed JSON

Deferred:

• Pre-retrieval SQL gate via queryd (the actual hybrid). The post-retrieval filter is an MVP that helps when the candidate set is mostly relevant; for aggressive filters that drop most results, a SQL pre-filter into matrix retrieval would surface the right candidates with less wasted embedding work.
• Filter language richer than equality (e.g. range, prefix, regex).

Acceptance gates:

• G3.6.A — MetadataFilter: {"state": "IL"} against a mixed-state corpus drops every non-IL result; MetadataFilterDropped reports the count.
• G3.6.B — List filter {"state": ["IL", "WI"]} keeps both states, drops the rest (OR within key).
• G3.6.C — Multi-key filter is AND: a result missing any key is dropped, no exception.

§3.7 — Operational rating wiring

What it is. PRD loop 4 (rating + distillation) needs real inflows to be a learning system rather than a substrate. The playbook-record endpoint (06e7152) takes one (query, answer, score) per call; productizing it into actual signal sources is what makes the system get smarter with use.

What's shipped (commit 6392772):

• POST /v1/matrix/playbooks/bulk — bulk-record N successes; per-entry success/failure response so callers can see which of a 4,701-row historical placement import succeeded vs which failed validation.
• Single-record path from 06e7152 unchanged.

Deferred:

• UI shim for click-tracking (no Go demo UI yet — the Bun demo at devop.live/lakehouse/ is still serving the public surface). When the Go UI lands or a feedback API is added to the Bun UI, every coordinator click → bulk-batched POST → playbook entry.
• Negative feedback (this match didn't work). Currently only positive scores are recorded; a rejection signal would help the learning loop avoid pushing bad matches.
• Time-decay on playbook scores so stale recommendations attenuate.

Acceptance gates:

• G3.7.A — Bulk POST of N entries returns {recorded, failed, results[]} with per-entry IDs/errors, no single-entry failure aborting the batch.
• G3.7.B — Each recorded entry surfaces in /v1/matrix/search with use_playbook=true after a re-query.

§3.8 — Observer-KB workflow runner (Archon-style multi-pass)

What it is. The architectural pattern documented in the Rust observer-kb branch (10 commits ahead of main, never merged) and proven by /home/profit/external/Archon's workflow engine. Multiple mode passes processing data, with each pass an objective measurement that contributes to the KB:

Raw data
   ↓ Mode: EXTRACT       structured facts/entities/relationships
   ↓ Mode: VALIDATOR     fact-check, confidence 1-10
   ↓ Mode: HALLUCINATION verify each claim, flag likely fabrications
   ↓ Mode: CONSENSUS     multiple passes until extraction converges
   ↓ Mode: REDTEAM       attack what survived, patch what fails
   ↓ Mode: PIPELINE      clean → Q&A structure → topic group → rank
   ↓ RENDER              curated doc anchored on questions

This is the orchestrator missing from §3.4 components 1-5: each SPEC §3.4 piece (relevance, downgrade, scorer, drift) is a "mode"; what's missing is the workflow engine that chains them.

Why it matters. Per the PRD's product vision: the observer should make actionable decisions based on watching what's successful. The workflow runner is how observers compose modes into multi-pass pipelines that score outcomes rigorously enough to feed the KB and inform the playbook substrate.

Reference materials on the system:

• /home/profit/lakehouse/.archon/workflows/lakehouse-architect-review.yaml (committed 69919d9 in main) — proves Archon-via-Lakehouse works with a 3-node shape → weakness → improvement workflow
• /home/profit/external/Archon — the upstream workflow engine (cloned 2026-04-26); packages/providers/src/community/pi/provider.ts has the local Lakehouse-routing mod committed locally as 3f2afc8 (not pushed to upstream coleam00/Archon)
• Rust observer-kb branch (10 commits, +4338/-55506 LoC) — apps/observer-kb/docs/PRD.md documents the multi-pass architecture; scripts/{deep_analysis,extract_knowledge,process_knowledge}.py are the Python prototypes that proved it on real ChatGPT/Claude PDF data (496 topics, 300 decisions, 100 insights extracted)

Components to port (in dependency order):

1. Workflow definition (internal/workflow/types.go) — YAML schema matching Archon's shape: name, description, provider, model, list of nodes each with id, prompt, allowed_tools, effort, idle_timeout, depends_on. The depends_on edges form a DAG; the runner resolves topologically.

2. Node executor (internal/workflow/runner.go) — given a workflow and a starting context, walks the DAG, executes each node by dispatching to the configured backend (matrix.Search, distillation.ScoreRecord, drift.ComputeScorerDrift, or a generic prompt-against-LLM via gateway /v1/chat), captures per-node output, makes it available as $<node_id>.output in subsequent nodes.

3. Provenance recording — every node execution lands an ObservedOp (via the observerd substrate from bc9ab93) with source: "workflow", the workflow name + node ID, input/output summaries, and timing. The ring buffer + JSONL log become the substrate for the rating+distillation loop's KB feed.

4. Mode catalog (internal/workflow/modes.go) — registry of the modes the runner can dispatch to. Each mode is a Go function matching a uniform func(ctx, input map[string]any) (map[string]any, error) signature so workflows can compose them. Initial modes from §3.4: matrix.search, matrix.relevance, matrix.downgrade, playbook.record, playbook.lookup, distillation.score, drift.scorer. Plus llm.chat for free-form mode prompts.

5. HTTP surface — POST /v1/observer/workflow/run accepts a workflow YAML body + a starting context; returns the per-node results + the chain of ObservedOps generated. GET /v1/observer/workflow/list lists workflows in a known directory for operator discoverability.

Why integrate into observerd, not a new service. The observer is the system resource that watches and records. Workflows ARE observation patterns — multi-step processes whose every step is recorded. Putting the runner inside observerd keeps the "measurement → KB feed" wiring tight; a separate service would re-implement the recording layer.

Acceptance gates:

• G3.8.A — Load a workflow YAML matching the Archon lakehouse-architect-review.yaml shape; runner executes the 3-node DAG topologically.
• G3.8.B — Each node execution lands an ObservedOp with source: "workflow" and the node's input/output. Stats endpoint shows the workflow ops.
• G3.8.C — A node referencing $<prior_node>.output in its prompt resolves correctly; missing reference is a clear error not a silent empty string.
• G3.8.D — Mode catalog dispatches matrix.search invocation to the matrixd backend without going through HTTP (in-process function call when matrixd is co-resident).

Status: PORT TARGET, not yet started. SPEC commits the design; implementation is its own wave (estimated L effort given the DAG runner + mode dispatch + provenance recording).

§3.9 — chatd (multi-provider LLM dispatcher) — SHIPPED 2026-04-30

Status: done at commit 05273ac (Phase 4 wave) + scrum-hardened at 0efc736. Composite port: 1,624 LoC, 19+ tests, 6/6 chatd_smoke.

What: cmd/chatd on :3220 routes POST /chat to a provider selected by model-name prefix or :cloud suffix:

ollama/<m>            → local Ollama at :11434 (no auth)
ollama_cloud/<m>      → ollama.com /api/generate (Bearer)
<m>:cloud             → ollama_cloud (suffix variant)
openrouter/<v>/<m>    → openrouter.ai (OpenAI-compat, Bearer)
opencode/<m>          → opencode.ai/zen/v1 (OpenAI-compat, Bearer)
kimi/<m>              → api.kimi.com/coding/v1 (OpenAI-compat, Bearer)
bare names            → ollama (default)

Provider key resolution: env var first (OPENROUTER_API_KEY, OPENCODE_API_KEY, KIMI_API_KEY, OLLAMA_CLOUD_KEY); then /etc/lakehouse/<provider>.env fallback (mode 0600). Empty key → provider stays unregistered (404 at first call instead of 503).

Companion to the model tier registry in lakehouse.toml [models] (local_fast / local_judge / cloud_judge / frontier_review / etc.), which maps tier names to model IDs. Callers reference cfg.Models.LocalJudge instead of literal strings. Bumping a tier is a 1-line config edit.

Quirks captured by today's scrum:

• Request.Temperature is *float64 (pointer) — Anthropic 4.7 (via OpenCode) rejects the field entirely with "temperature is deprecated." Pointer lets us omit when caller didn't set explicitly.
• Local Ollama defaults think=false — qwen3.5:latest is reasoning- capable but the inner-loop hot path wants direct answers, not reasoning traces consuming the token budget.

Replaces the Rust adapters:

• crates/gateway/src/v1/ollama_cloud.rs → internal/chat/ollama_cloud.go
• crates/gateway/src/v1/openrouter.rs → internal/chat/openai_compat.go (shared with opencode + kimi)
• crates/gateway/src/v1/opencode.rs → same shared helper
• crates/aibridge/* → internal/chat/ (cleaner abstraction)

Reusable downstream: scripts/scrum_review.sh runs a 3-lineage cross-review (Opus + Kimi + Qwen3-coder) by POST-ing each diff to chatd's /v1/chat. Same vehicle the harness's own scrum-hardening used to find its own bugs.

§3.10 — local-review-harness (sibling tool, separate repo)

Where: git.agentview.dev/profit/local-review-harness (also SMB-mounted at /home/profit/share/local-review-harness-full-md/).

What: a local-first code review harness — walks a target repo, runs evidence-bearing static checks (12 analyzers covering hardcoded paths, raw SQL, wildcard CORS, secrets, exec/spawn, large files, TODO/FIXME, missing tests, .env files committed, exposed mutation endpoints, hardcoded private IPs), produces Scrum-style markdown reports + JSON receipts. No cloud deps. Single static Go binary.

Status: Phase A (skeleton) + Phase B (MVP — static-only path) shipped at first commit f3ee472. 5 acceptance gates green plus self-review (the harness reviews its own repo).

Phases C–E pending: local-Ollama LLM review, validation cross- check, append-only .memory/, diff/rules subcommands.

Why a sibling tool, not a Lakehouse module: PROMPT.md "Strategic Goal" — the harness eventually plugs into OpenClaw / MCP tools / Lakehouse memory / playbook sealing / observer review loop. But first it has to be reliable, inspectable, and evidence-driven on its own. Lakehouse-Go integration is post-Phase-E (probably E+1: write the harness's findings into Lakehouse's playbook substrate via /v1/matrix/playbooks/record).

Cross-pollination opportunities (post-MVP):

• Replace internal/llm/ollama.go in the harness with a thin client pointed at chatd's /v1/chat — frontier judges become a config toggle.
• Feed harness findings into Lakehouse-Go's pathway memory as a drift signal (which static checks fired this run vs last).
• Use the harness's .memory/known-risks.json as a corpus the matrix indexer can retrieve from when the same risk pattern appears.

§3.3 — UI (HTMX)

Approach: server-rendered Go templates using html/template, HTMX for partial-page swaps, Alpine.js for client-side interactivity where needed. Single binary serves API + UI.

Acceptance gates:

• G3.3.A — Ask tab: type natural-language question, get answer from RAG endpoint, render in-page without full reload
• G3.3.B — Explore tab: paginated dataset list with hot-swap badge rendering
• G3.3.C — SQL tab: textarea → submit → tabular result rendered in-page
• G3.3.D — System tab: live tail of /storage/errors and /hnsw/trials via HTMX polling

Fallback if HTMX feels limiting: split repo golangLAKEHOUSE-ui with Vite + React, served as static files by Go gateway. Costs an extra repo + build chain.

§3.4 — Pathway memory port

Constraint: the Rust pathway_memory and TS implementations were byte-matching by ADR-021. The byte contract was verified by running both implementations on the same input tokens and asserting matching bucket indices.

Go port plan:

• Port the 32-bucket SHA256-keyed token hash exactly. Verify on a golden input that Go produces the same bucket vector as Rust.
• Port the JSON state file format verbatim — the existing 88 traces in data/_pathway_memory/state.json reload as-is into the Go implementation.
• Port the matrix-correctness layer (ADR-021's SemanticFlag, BugFingerprint, TypeHint) — these are pure value types, trivially portable.

Acceptance gates:

• G3.4.A — Load existing state.json, run replay on the same 11 prior successful pathways, all 11 succeed (matching the Rust 11/11 baseline).
• G3.4.B — Bucket vector for a fixed test input byte-matches the Rust output.

---

§4. Phase plan

Phase G0 — Skeleton (Week 1–3)

Scope: smallest end-to-end ingest + query path working in Go.

Component	Deliverable
cmd/gateway	HTTP on :3100, /health, /v1/chat proxy stub
cmd/catalogd	In-memory registry + Parquet manifest persistence
cmd/storaged	Single-bucket S3 / local FS, no error journal yet
cmd/ingestd	CSV → Parquet, schema inference, register-on-ingest
cmd/queryd	DuckDB-backed POST /sql endpoint

Acceptance: upload a CSV via POST /ingest, query it via POST /sql with a SELECT, get rows back. Single-bucket. No vector, no profile, no UI.

Phase G1 — Vector + RAG (Week 4–6)

Component	Deliverable
cmd/vectord	Embed-on-ingest (calls Python sidecar), HNSW build, POST /search
cmd/gateway	Add POST /rag (embed → search → retrieve → generate via aibridge)
cmd/aibridge	HTTP client to existing Python sidecar

Acceptance: ingest 15K resumes (the original Phase 7 fixture), ask "find me a forklift operator with OSHA-10 in IL", get ranked results with LLM-generated explanation grounded in the retrieved chunks.

Phase G2 — Federation + profiles (Week 7–8)

Component	Deliverable
cmd/storaged	Multi-bucket registry, rescue bucket, error journal at primary://_errors/
Profile system	Per-reader profile bound to bucket + vector index
Hot-swap	Atomic pointer swap for index generations

Acceptance: two profiles bound to two buckets, queries scoped correctly, hot-swap a vector index without query interruption, rollback works.

Phase G3 — Pathway memory + distillation (Week 9–11)

Component	Deliverable
cmd/vectord	Pathway memory module ported, 88 traces reloaded
Distillation pipeline	SFT export, contamination firewall, scorer
Audit baselines	audit_baselines.jsonl longitudinal signal port

Acceptance: replay 11 prior successful pathways, all 11 succeed. Re-run distillation acceptance on the frozen fixture set, 22/22 pass.

Phase G4 — TS surfaces → Go (Week 12–14)

Component	Deliverable
cmd/mcp	MCP server (replaces Bun) — /v1/chat, intelligence endpoints
cmd/observer	Autonomous iteration loop, op recording
cmd/auditor	PR audit pipeline (kimi/haiku/opus rotation)
cmd/scrum	Scrum master pipeline (replaces TS)

Acceptance: open a test PR, auditor cycles within 90s, emits verdict to data/_auditor/kimi_verdicts/, behavior matches Rust+TS era within tolerance.

Phase G5 — UI + demo parity (Week 15–16)

Component	Deliverable
cmd/gateway	Serves HTMX templates + static demo HTML
Demo at devop.live/lakehouse/	Parity with current Bun demo
Staffer console at /console	Parity

Acceptance: devop.live/lakehouse/ cuts over from Bun to Go gateway. Section ① / ② / ③ all render. Compact contract cards still expand with Project Index. Fill-probability bars still paint.

---

§5. Repo layout

golangLAKEHOUSE/
├── docs/
│   ├── PRD.md                    ← this PRD
│   ├── SPEC.md                   ← this spec
│   ├── DECISIONS.md              ← Go-era ADRs (start fresh, reference Rust ADRs by number)
│   └── ADR-XXX-*.md              ← per-ADR detail
├── cmd/
│   ├── gateway/                  ← main HTTP/gRPC ingress
│   ├── catalogd/
│   ├── storaged/
│   ├── queryd/
│   ├── ingestd/
│   ├── vectord/
│   ├── journald/
│   ├── mcp/
│   ├── observer/
│   ├── auditor/
│   └── scrum/
├── internal/                     ← shared packages, not exported
│   ├── aibridge/
│   ├── validator/
│   ├── truth/
│   ├── shared/
│   ├── proto/                    ← generated protobuf
│   └── pathway/
├── pkg/                          ← public Go packages (none initially)
├── web/                          ← UI (HTMX templates + static)
│   ├── templates/
│   └── static/
├── scripts/                      ← cold-start, smoke, distill scripts
├── tests/                        ← golden files, integration tests
├── go.mod
├── go.sum
└── README.md

Single Go module. All commands and internal packages live under golangLAKEHOUSE/. No nested modules unless a package needs an independent release cadence (none expected).

Build: go build ./cmd/... produces all binaries.

---

§6. Migration data plan

What ports verbatim

• Parquet datasets at data/datasets/*.parquet — read by Go directly.
• Catalog manifests — Parquet, ports as data not code.
• Pathway memory state — JSON, ports if §3.4 byte-matching gate passes.

What rebuilds

• HNSW indexes — rebuild from Parquet embeddings on first Go startup.
• Auditor verdicts on PRs — old PRs won't be re-audited; lineage starts fresh on the new repo's PRs.

What's archived

• The Rust crates/ tree — preserved in the original repo at the cutover commit, tagged pre-go-rewrite-2026-04-28 for reference.
• TS surfaces (mcp-server/, auditor/, etc.) — preserved in the original repo at the same tag.
• Distillation v1.0.0 substrate (tag distillation-v1.0.0, e7636f2) — kept as the historical reference; Go re-implementation ports the LOGIC but not the bit-identical-reproducibility property unless an ADR re-establishes it.

What's discarded

• crates/vectord-lance/ (Lance backend, see PRD §Hard problems §2)
• crates/lance-bench/ (criterion benchmarks specific to Lance)

---

§7. Acceptance: when is the rewrite done?

The Go Lakehouse reaches feature parity when:

1. All 12 Rust PRD invariants hold (object-storage source of truth, catalog metadata authority, idempotent ingest, hot-swap atomicity, profiles, etc.).
2. The 16 distillation acceptance gates pass (re-run ./scripts/distill audit-full against the Go pipeline).
3. The 22/22 acceptance fixtures from tests/fixtures/distillation/ acceptance/ pass under the Go implementation.
4. The 145 unit tests of distillation v1.0.0 are ported and pass.
5. devop.live/lakehouse/ demo cuts over to Go gateway with no visible UI regressions.
6. Auditor emits Kimi/Haiku/Opus verdicts on a test PR, matching the cross-lineage rotation behavior.
7. The 88 pathway traces replay with 11/11 prior successes reproduced.

At that point the Rust repo enters maintenance-only mode (security fixes), and the Go repo becomes the live system.

---

§8. Ratified — Phase G0 unblocked (2026-04-28, J)

#	Decision	Spec impact
1	DuckDB via cgo (marcboeker/go-duckdb)	§3.1 option A — proceed
2	HTMX + html/template + Alpine.js	§3.3 option A — proceed
3	git.agentview.dev/profit/golangLAKEHOUSE	repo location locked
4	Distillation rebuilt in Go (no bit-identical port)	§6 — port logic, not fixtures
5	Pathway memory starts empty; old traces noted	§3.4 G3.4.A is now "build initial state from scratch in Phase G3"; G3.4.B (byte-match) preserved as the porting correctness gate when the algorithm is reimplemented
6	Auditor longitudinal signal restarts	new audit_baselines.jsonl lineage starts on first Go-era PR

See docs/DECISIONS.md ADR-001 for full rationale and docs/RUST_PATHWAY_MEMORY_NOTE.md for where the legacy 88 traces live.

Phase G0 is now unblocked. Next step: bootstrap the Go module skeleton + push to Gitea, then begin §4 Phase G0 implementation.