profit/lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
lakehouse/docs/PHASES.md
root 239e471223 Phase 3: AI integration with Ollama via Python sidecar 
- sidecar: FastAPI app with /embed, /generate, /rerank hitting Ollama
- sidecar: Dockerfile, env var config (EMBED_MODEL, GEN_MODEL, RERANK_MODEL)
- aibridge: reqwest HTTP client with typed request/response structs
- aibridge: Axum proxy endpoints (POST /ai/embed, /ai/generate, /ai/rerank)
- gateway: wires AiClient with SIDECAR_URL env var
- e2e verified: nomic-embed-text returns 768d vectors, qwen2.5 generates text

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 05:53:56 -05:00

2.6 KiB
Raw Blame History

Phase Tracker

Phase 0: Bootstrap

  • 0.1 — Cargo workspace with all crate stubs compiling
  • 0.2 — shared crate: error types, ObjectRef, DatasetId
  • 0.3 — gateway with Axum: GET /health → 200
  • 0.4 — tracing + tracing-subscriber wired in gateway
  • 0.5 — justfile with build, test, run recipes
  • 0.6 — docs committed to git

Gate: PASSED — All crates compile. Gateway runs. Logs emit. Docs committed.

Phase 1: Storage + Catalog

  • 1.1 — storaged: object_store backend init (LocalFileSystem)
  • 1.2 — storaged: Axum endpoints (PUT/GET/DELETE/LIST /objects/{key})
  • 1.3 — shared/arrow_helpers.rs: RecordBatch ↔ Parquet + schema fingerprinting
  • 1.4 — catalogd/registry.rs: in-memory index + manifest persistence to object storage
  • 1.5 — catalogd/schema.rs: schema fingerprinting (merged into shared/arrow_helpers.rs)
  • 1.6 — catalogd service: POST/GET /datasets + GET /datasets/by-name/{name}
  • 1.7 — gateway routes to storaged + catalogd with shared state

Gate: PASSED — PUT object → register dataset → list → get by name. All via gateway HTTP.

Phase 2: Query Engine

  • 2.1 — queryd: SessionContext + object_store config (custom scheme to avoid path doubling)
  • 2.2 — queryd: ListingTable from catalog ObjectRefs with schema inference
  • 2.3 — queryd service: POST /query/sql → JSON (columns + rows + row_count)
  • 2.4 — queryd → catalogd wiring (reads dataset list, registers as tables)
  • 2.5 — gateway routes /query with QueryEngine state

Gate: PASSED — SELECT *, WHERE/ORDER BY, COUNT/AVG all return correct results via catalog.

Phase 3: AI Integration

  • 3.1 — Python sidecar: FastAPI + Ollama (embed/generate/rerank) — real models, no mocks
  • 3.2 — Dockerfile for sidecar
  • 3.3 — aibridge/client.rs: reqwest HTTP client with 120s timeout
  • 3.4 — aibridge service: Axum proxy endpoints (POST /ai/embed, /ai/generate, /ai/rerank)
  • 3.5 — Model config via env vars (EMBED_MODEL, GEN_MODEL, RERANK_MODEL, SIDECAR_URL)

Gate: PASSED — Gateway → aibridge → sidecar → Ollama → real 768d embeddings + generation.

Phase 4: Frontend

  • 4.1 — Dioxus scaffold, WASM build
  • 4.2 — Dataset browser
  • 4.3 — Query editor + results table
  • 4.4 — Error display + loading states

Gate: Browse datasets and query from browser.

Phase 5: Hardening

  • 5.1 — Proto definitions
  • 5.2 — Internal gRPC migration
  • 5.3 — OpenTelemetry tracing
  • 5.4 — Auth middleware
  • 5.5 — Config-driven startup

Gate: gRPC internals, traces, auth, restartable from repo + config.