lakehouse/docs/EXECUTION_PLAN.md

Execution Plan — Phases B through E

Created: 2026-04-16 Status: Active planning document — update as phases complete or scope shifts Owner: J

This plan sequences the work J and Claw agreed on during the 2026-04-16 reframe session, after stress-testing the "dual-use substrate" vision and aligning it with llms3.com's architectural patterns.

---

The four phases, at a glance

Phase	Work	Prereq	Estimated cost	Risk
B	Lance pilot on one vector index	None	1 focused session	Medium — new dep, unfamiliar surface
C	Decoupled embedding refresh pipeline	Benefits from B's outcome	1 focused session	Low — additive, doesn't break existing
D	AI-safe views	Phase 13 (done)	1 focused session	Low — builds on existing catalog + tool registry
E	Soft deletes / tombstones	None	1-2 focused sessions	Medium — touches query path, compaction

Each "focused session" ≈ 3-4 hours of coding + verification + doc update.

---

Phase B — Lance pilot (the storage format question)

Why now

J's LLMS3 knowledge base explicitly positions Lance as alternative_to Parquet for vector workloads. We admitted in the 2026-04-16 stress test that our Parquet-portability argument for vectors is weaker than advertised — our vector Parquet blobs aren't readable by DuckDB/Polars anyway. Lance could unlock: random-row access, disk-resident indexes, time-travel, better compression. Or it could disappoint and we lock in Parquet with a written reason why.

Commit to one answer backed by measurements. No ambiguity after this phase.

Scope

1. Add lance crate as a dep in vectord, behind a feature = "lance" flag initially so we can build without forcing users to install it
2. New module vectord::lance_store — mirror of vectord::store but against Lance format
3. New endpoint POST /vectors/lance/index — build a Lance index for a named source (parallel to the existing Parquet path)
4. Benchmark script that runs against resumes_100k_v2 (the existing 100K reference)

Measured dimensions

All benchmarks against resumes_100k_v2 (100K × 768d), cold start:

Metric	Parquet baseline	Lance	Threshold to migrate
Cold load from disk	~2.8s (measured)	?	≥2× faster
Search latency p50	873us (ec=80 es=30)	?	Within 50%
Disk size	330MB	?	Comparable or better
Single-row random access	N/A (full scan)	?	<10ms
Append 10K new rows	Full rewrite (~3s)	?	Incremental <500ms

Decision rules

• Lance wins cold-load by ≥2× AND matches search latency: migrate vector storage to Lance. Dataset tables stay Parquet. Update ADR-008 → ADR-019.
• Lance within 50% across board: stay Parquet. Document ceiling honestly in PRD (already done). Lance revisit when we have a problem Parquet can't solve.
• Lance loses: close the door, don't revisit without new evidence. Write ADR-019 as "Lance evaluated, rejected, here's why with numbers."

Success gate

Benchmark output table posted to docs/ADR-019-vector-storage.md with measured numbers in each cell of the table above. Decision rule applied mechanically. No "let's defer the call."

Rollback

The feature = "lance" flag means if the pilot goes badly, cargo build without the flag is unchanged. No production migration happens until ADR-019 commits to the change. Safe experiment.

---

Phase C — Decoupled embedding refresh

Why now

llms3.com's lakehouse architecture explicitly separates "transactional data mutations" from "asynchronous vector refresh cycles." Today we couple them — an ingest writes rows AND embeddings in one flow. That means:

• Adding 1K rows to a 100K-row dataset forces re-embedding of ALL rows (or nothing)
• No notion of "embeddings are stale, schedule a refresh tonight"
• The embedding cost (Ollama-bound, the bottleneck) is synchronous with ingest

Scope

1. Add fields to DatasetManifest:
   • last_embedded_at: Option<DateTime>
   • embedding_stale_since: Option<DateTime> (set when data written but embeddings not refreshed)
   • embedding_refresh_policy: RefreshPolicy — Manual | OnAppend | Scheduled(cron)
2. Decouple ingest from embed: ingest writes data + marks embeddings stale; embedding runs separately
3. New endpoint: POST /vectors/refresh/{dataset} — diffs existing vectors vs current rows, only embeds new/changed (keyed by doc_id)
4. Background scheduler (or cron trigger) — for Scheduled policy, re-runs refresh per schedule
5. GET /vectors/stale — lists datasets with stale embeddings and how stale

Measured success

• Ingest a 1K-row append to kb_team_runs (currently 586 rows, Postgres-sourced).
• Catalog shows embedding_stale_since = now.
• POST /vectors/refresh/kb_team_runs embeds only the 1K new rows, not all 1586.
• Result: new rows searchable, old embeddings unchanged, total Ollama time ~5s instead of ~30s.

Dependencies on Phase B

If Lance wins Phase B, this is dramatically easier — Lance supports native append. If we stay Parquet, we need a "vectors delta" Parquet file that merges at read time (same pattern as Phase 8's data delta files). ~100 extra LOC if we stay Parquet.

Rollback

The refresh_policy field defaults to Manual for all existing datasets, so no behavior change for anything already in the system. Opt-in per dataset.

---

Phase D — AI-safe views

Why now

llms3.com's framing: "AI-safe views enforcing row/column security + PII tokenization before model exposure." Phase 13 gave us role-based column masking at query time. That's per-query enforcement. "Views" means pre-materialized: create candidates_safe once, bind model X to that view, model X can never accidentally see raw candidates.

This is also the precondition for Phase 17 (model profiles) to be meaningfully safe. "Bind model to dataset" isn't enough — needs to be "bind model to a safe view of the dataset."

Scope

1. New catalog entity: AiView with fields name, base_dataset, columns[] (whitelist), row_filter (optional SQL WHERE clause), column_redactions[] (PII tokenization rules)
2. Persistence: _catalog/views/{name}.json alongside manifests
3. Query-rewrite layer: when a query references candidates_safe, DataFusion sees an equivalent SELECT (whitelisted cols) FROM candidates WHERE (row_filter) — with redactions applied as expressions
4. Endpoint: POST /catalog/views to create, GET /catalog/views to list, GET /catalog/views/{name}/preview to see what it looks like
5. Tool registry integration: tools can bind to an AiView instead of a raw table; agent invocations go through the view

Measured success

• Create view candidates_safe = SELECT candidate_id, skills, city FROM candidates WHERE status != 'blocked'.
• Agent (tool registry) calls search_candidates bound to candidates_safe.
• Agent cannot see email, phone, ssn, or status='blocked' rows, even if it writes raw SQL.
• Audit log records agent accessed candidates_safe, not candidates.

Dependencies

• Phase 13 already provides the sensitivity classification layer
• Phase 12 tool registry already exists
• This phase is the bridge between them for agent access

Rollback

Views are additive. Dropping the feature = delete view definitions, tool registry falls back to direct table access. No data migration needed.

---

Phase E — Soft deletes / tombstones

Why now

GDPR/CCPA compliance for staffing data. Today, ops::delete physically deletes a parquet object — fine for whole datasets, useless for "delete one candidate's record." To delete one row we'd have to rewrite the whole candidates.parquet which at 100K rows is 10MB of churn per deletion.

llms3.com lists "deletion vectors" as a core lakehouse pattern (Iceberg/Delta/Hudi all implement it). This is the single biggest compliance gap in the current system.

Scope

1. New sidecar per dataset: {dataset}_tombstones.parquet with columns {row_key, deleted_at, actor, reason}
2. Delete API: POST /catalog/datasets/{name}/tombstone with {row_keys[], reason, actor}
3. Query-time filter: queryd automatically LEFT JOINs tombstones and filters out deleted rows
4. Compaction integration: Phase 8 compaction reads base + delta + tombstones, writes a clean base without tombstoned rows, clears the tombstone sidecar
5. Event journal integration (Phase 9): every tombstone emits a journal event with full context

Measured success

• POST /catalog/datasets/candidates/tombstone with {row_keys: ["CAND-123"], reason: "GDPR request", actor: "legal@company"}
• SELECT COUNT(*) FROM candidates drops by 1 immediately
• SELECT * FROM candidates WHERE candidate_id = 'CAND-123' returns empty
• GET /journal/history/CAND-123 shows the tombstone event
• After scheduled compaction, the tombstone is materialized — candidates.parquet no longer contains CAND-123, tombstone sidecar is emptied for that row key

Dependencies

• Phase 8 delta/merge-on-read pattern (done) — tombstones are a third layer at read time
• Phase 9 event journal (done) — tombstones emit journal events

Rollback

If query rewrite becomes too complex, fallback: tombstones stored but applied only during compaction (not at query time). Queries return deleted rows until compaction runs. Less useful but safer.

---

Cross-phase concerns

Phases that need federation layer 2 (task #5)

Every phase above assumes the federation foundation (shipped 2026-04-16) but NOT federation layer 2 (cross-bucket SQL, profile activation, X-Lakehouse-Bucket header).

Implication: Phases B-E can proceed on the primary bucket without blocking on federation layer 2. Federation layer 2 becomes valuable when we want multi-profile model scoping (Phase 17). Sequence:

A (done) → B → C (+D in parallel) → federation layer 2 → Phase 16 → Phase 17 → E

Phases that need federation layer 2 FIRST

None of B/C/D/E strictly need it. Phase 16 (hot-swap) benefits from it. Phase 17 (model profiles) depends on it heavily.

What NOT to build in B-E

• Distributed query — wait for a real scale problem
• Replacement of DataFusion — working fine, stay put
• Iceberg/Delta Lake migration — explicitly out of scope per ADR-009
• Live streaming / CDC — explicit non-goal

Definition of done for each phase

Each phase completes when:

1. Code shipped and building clean
2. Success gate measurably passed
3. Relevant ADR added to docs/DECISIONS.md (or updated)
4. docs/PHASES.md checkbox flipped with measurement data
5. PRD invariants checked — if a new invariant emerged, add it
6. One regression test in the crate or HTTP integration test

---

Session plan (what to do in what order)

Next session

1. Phase B — Lance pilot. Single session. Answers the biggest open architectural question.
2. Based on outcome, write ADR-019 with the decision + data.
3. Update PHASES.md with Phase 18 status (Lance evaluated).

Session after

4. Phase C — Decoupled embedding refresh. Implementation shaped by B's outcome (append is easy on Lance, requires delta logic on Parquet).

Session after that

5. Federation layer 2 OR Phase D (AI-safe views) — J decides based on priority. Federation layer 2 unlocks model profiles (Phase 17); AI-safe views is standalone value.

Final session for this track

6. Phase E — Soft deletes. The compliance-driven phase. Fits cleanly after everything else because it touches the query path and wants to be built after query optimizations stabilize.

Milestone checkpoint

After Phase E, stop and reassess. We'll have:

• Lance decision made and committed
• Decoupled embedding pipeline
• AI-safe view enforcement
• Soft delete semantics

That's a substantial capability increase. Plausible "pause, write a retrospective, decide on Phase 16/17" moment.