01373c0e45
- proto: lakehouse.proto with CatalogService, QueryService, StorageService, AiService - proto crate: tonic-build codegen from proto definitions - catalogd: gRPC CatalogService implementation - gateway: dual HTTP (:3100) + gRPC (:3101) servers - gateway: OpenTelemetry tracing with stdout exporter - gateway: API key auth middleware (toggleable) - shared: TOML config system with typed structs and defaults - lakehouse.toml config file - ADR-006 and ADR-007 documented Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
37 lines
2.2 KiB
Markdown
37 lines
2.2 KiB
Markdown
# Architecture Decision Records
|
|
|
|
## ADR-001: Object storage as source of truth
|
|
**Date:** 2026-03-27
|
|
**Decision:** All data lives in S3-compatible object storage. No traditional database.
|
|
**Rationale:** Eliminates DB operational overhead, enables infinite scale at storage tier, forces clean separation of data and metadata.
|
|
|
|
## ADR-002: Catalog metadata persistence
|
|
**Date:** 2026-03-27
|
|
**Decision:** catalogd persists manifests as Parquet files in object storage. In-memory index rebuilt on startup.
|
|
**Rationale:** No external DB dependency. Storage is already the source of truth. Write-ahead pattern ensures consistency.
|
|
|
|
## ADR-003: Real models only (no mocks)
|
|
**Date:** 2026-03-27
|
|
**Decision:** AI sidecar hits Ollama with real models from Phase 3 onward. No stub/mock endpoints.
|
|
**Rationale:** Local Ollama instance available with nomic-embed-text, qwen2.5, mistral, gemma2, llama3.2. Mocks hide integration bugs.
|
|
|
|
## ADR-004: Python sidecar as Ollama adapter
|
|
**Date:** 2026-03-27
|
|
**Decision:** Python FastAPI sidecar is a thin HTTP adapter over Ollama's API. No model loading in Python.
|
|
**Rationale:** Ollama handles model lifecycle, GPU scheduling, caching. Sidecar stays stateless and lightweight — no torch/transformers deps.
|
|
|
|
## ADR-005: HTTP-first, gRPC later
|
|
**Date:** 2026-03-27
|
|
**Decision:** All inter-service communication uses HTTP through Phase 4. gRPC migration in Phase 5.
|
|
**Rationale:** HTTP is simpler to debug, test, and iterate on. gRPC adds protobuf compilation and streaming complexity before APIs stabilize.
|
|
|
|
## ADR-006: TOML config over environment variables
|
|
**Date:** 2026-03-27
|
|
**Decision:** System configuration via lakehouse.toml file, with sane defaults for all values.
|
|
**Rationale:** Config files are versionable, self-documenting, and support structured data. Env vars remain available as overrides. System must be restartable from repo + config alone.
|
|
|
|
## ADR-007: Dual HTTP+gRPC on gateway
|
|
**Date:** 2026-03-27
|
|
**Decision:** Gateway serves HTTP on :3100 (external) and gRPC on :3101 (internal). Both run in the same process.
|
|
**Rationale:** Single binary simplifies deployment. HTTP stays for browser/curl access. gRPC provides typed contracts for service-to-service calls. No premature microservice split.
|