lakehouse/docs/PRD.md

PRD: Lakehouse — Rust-First Object Storage System

Status: Active Created: 2026-03-27 Owner: J

---

Problem

Traditional data platforms couple storage, compute, and metadata into monolithic databases. This creates vendor lock-in, scaling bottlenecks, and opaque data access. AI workloads bolt onto these systems awkwardly, sharing resources with transactional queries.

We need a system where:

• Object storage is the source of truth (not a database)
• Metadata, access, and execution are controlled by Rust services
• Queries run directly over object storage via Arrow/Parquet
• AI inference is isolated and swappable
• The entire system is rebuildable from repository + docs alone

---

Solution

A modular Rust service mesh over S3-compatible object storage.

Locked Stack

Layer	Technology	Locked
Frontend	Dioxus	Yes
API	Axum + Tokio	Yes
Object Storage Interface	Apache Arrow object_store	Yes
Storage Backend	RustFS (fallback: SeaweedFS)	Yes
Query Engine	DataFusion	Yes
Data Format	Parquet + Arrow	Yes
RPC (internal)	tonic (gRPC)	Yes
AI Runtime	Ollama (local models)	Yes
AI Boundary	Python FastAPI sidecar → Ollama HTTP API	Yes

No new frameworks. No exceptions.

---

Architecture

Services

Service	Responsibility
gateway	HTTP ingress, routing, auth envelope, middleware
catalogd	Metadata control plane — dataset registry, schema versions, manifest index
storaged	Object I/O — read/write/list/delete via object_store crate
queryd	SQL execution — DataFusion over registered Parquet datasets
aibridge	Rust↔Python boundary — HTTP client to FastAPI sidecar
ui	Dioxus frontend — dataset browser, query editor, results viewer
shared	Types, errors, Arrow helpers, protobuf definitions

AI Sidecar

Python FastAPI process that adapts Ollama's HTTP API into Arrow-compatible formats:

• POST /embed → nomic-embed-text via Ollama
• POST /generate → configurable model (qwen2.5, mistral, gemma2, llama3.2)
• POST /rerank → cross-encoder reranking via generate endpoint

No mocks. No stubs. Real models from day one. Ollama manages model lifecycle, GPU scheduling, caching. Sidecar is stateless passthrough.

Data Flow

Client → gateway → catalogd (metadata lookup)
                  → storaged (object read/write)
                  → queryd  (SQL execution over Parquet)
                  → aibridge → sidecar → Ollama (inference)

Invariants

1. Object storage = source of truth for all data
2. catalogd = sole metadata authority (datasets, schemas, manifests)
3. No raw data stored in catalog — only pointers (bucket, key, schema fingerprint)
4. storaged never interprets data — dumb pipe with presigned URLs
5. queryd registers tables via catalog pointers, not by scanning storage
6. aibridge is stateless — Python sidecar is replaceable without touching Rust
7. All services are modular and independently replaceable

Dependency Graph

shared ← storaged ← catalogd ← queryd
shared ← aibridge
gateway → {storaged, catalogd, queryd, aibridge}
ui → gateway (HTTP only, no crate dependency)

---

Phases

Phase 0: Bootstrap

Workspace compiles, gateway serves health check, structured logging works.

Gate: cargo build clean, GET /health returns 200, logs on stdout, docs committed.

Phase 1: Storage + Catalog

Write Parquet to object storage, register in catalog, read back.

Gate: Upload Parquet → register dataset → retrieve metadata → read back. All via gateway HTTP.

Phase 2: Query Engine

SQL queries over registered Parquet datasets via DataFusion.

Gate: SELECT * FROM dataset LIMIT 10 returns correct results. Resolution goes through catalog.

Phase 3: AI Integration

Python sidecar with real Ollama models. Embeddings, generation, reranking.

Gate: Rust sends text → Python → Ollama → real embeddings return as Arrow-compatible floats.

Phase 4: Frontend

Dioxus UI: dataset browser, query editor, results table.

Gate: User can browse datasets and run queries from browser.

Phase 5: Hardening

gRPC internals, OpenTelemetry, auth, config-driven startup.

Gate: Services communicate via gRPC. Traces propagate. Auth enforced. System restartable from repo + config.

---

Available Local Models

Model	Use
nomic-embed-text	Embeddings (768d)
qwen2.5	Code generation, structured output
mistral	General generation
gemma2	General generation
llama3.2	General generation

Model selection via environment variables. No hardcoded model names in Rust code.

---

Non-Goals

• Multi-tenancy
• Streaming ingestion / CDC
• Custom file formats
• Query caching / materialized views
• Wrapping object_store with another abstraction
• Cloud deployment (local-first)

---

Risks

Risk	Severity	Mitigation
RustFS immaturity	High	Start with LocalFileSystem, test against MinIO, RustFS last. SeaweedFS fallback.
DataFusion table registration overhead	Medium	Lazy registration + LRU cache of SessionContext instances.
Catalog consistency without DB	Medium	Write-ahead: persist manifest before in-memory update. Rebuild from storage on restart.
Dioxus WASM gaps	Medium	Phase 4 is last. Fallback to plain HTML if blocked.
Schema evolution	Medium	Schema fingerprinting in Phase 1. Validate before query.

---

Operating Rules

1. PRD > architecture > phases > status > git
2. Git is memory, not chat
3. No undocumented changes
4. No silent architecture drift
5. Always work in smallest valid step
6. Always verify before moving on