lakehouse/docs/ADR-020-universal-id-mapping.md

ADR-020: Universal ID Mapping — the hybrid search identity problem

Status: Proposed — 2026-04-17 Triggered by: ID mismatch between vector doc_ids and SQL primary keys Owner: J

---

Problem

The hybrid search endpoint (POST /vectors/hybrid) filters SQL results by primary key, then matches those keys against vector doc_ids. This requires the two ID spaces to be compatible. Currently they're not:

Source	SQL primary key	Vector doc_id	Match?
ethereal_workers	worker_id = 4925	W-4925	Only with W- strip
workers_500k	worker_id = 41566	W500K-41566	Only with W500K- strip
candidates	candidate_id = CAND-055035	CAND-055035	Yes (same format)
workers_5k_proof	worker_id = 3200	W5K-3200	Only with W5K- strip

Every time a new dataset is ingested and embedded, a new prefix appears and the hybrid search breaks until someone hardcodes another strip_prefix line.

This violates PRD invariant: "Any data source can be ingested without pre-defined schemas." The ID format IS a pre-defined schema that the hybrid search depends on.

Root cause

When the embedding pipeline creates a vector index, it generates doc_ids by concatenating a prefix + the source row's ID column. The prefix is chosen at index creation time and baked into the Parquet vector file. The SQL dataset knows nothing about this prefix.

The hybrid search then has to reverse-engineer the prefix to match vector results against SQL rows. This is fragile and breaks on every new data source.

Solution: catalog-level ID mapping

Option A: Normalize doc_ids at embedding time (simplest)

When creating a vector index, don't prefix the doc_id. Use the raw value from the source column. If the source has worker_id = 4925, the vector doc_id is just "4925" — no W-, no W500K-.

Pros: Simplest. Hybrid search just compares strings directly. Cons: Doc_ids across different indexes could collide (two datasets both have worker_id=1). Need to scope by index name.

Option B: Catalog stores the mapping (most robust)

The catalog maintains a mapping table:

index_name | doc_id_prefix | source_dataset | source_id_column
-----------+---------------+----------------+-----------------
resumes    | CAND-         | candidates     | candidate_id
workers_v1 | W500K-        | workers_500k   | worker_id

Hybrid search reads this mapping and applies the prefix/strip automatically. New datasets register their mapping at index creation.

Pros: Handles all cases. Self-describing. No code changes for new data. Cons: One more lookup per hybrid search (trivial perf cost).

Option C: Pass the mapping in the request (pragmatic)

The hybrid search request already accepts id_column. Extend it with id_prefix so the caller says "vector doc_ids have prefix W500K- and the SQL column is worker_id."

{
  "index_name": "workers_500k_v1",
  "id_column": "worker_id",
  "id_prefix": "W500K-",
  "sql_filter": "role = 'Forklift Operator' AND state = 'IL'",
  ...
}

Pros: Zero backend change. Caller already knows the context. Cons: Caller has to know the prefix — not self-service.

Recommendation

Option A for new indexes, Option B for registry.

1. Change the embedding pipeline to use RAW IDs (no prefix) as the default. Existing indexes keep their prefixed IDs.
2. Add an id_prefix column to IndexMeta so the catalog knows how to map between vector and SQL IDs.
3. Hybrid search reads IndexMeta.id_prefix and applies it automatically.
4. Remove the hardcoded strip_prefix chain.

This means: ingest a new dataset, embed it, hybrid search works immediately. No code changes. The system is truly "any data source."

Implementation

1. IndexMeta gains id_prefix: Option<String> (default None = no prefix)
2. Embedding pipeline: when id_prefix is None, use raw IDs from source
3. Hybrid search: read IndexMeta.id_prefix, prepend to SQL IDs before matching
4. Migration: existing indexes retain their prefix in IndexMeta