lakehouse/docs/IDENTITY_SERVICE_DESIGN.md

Identity Service — Phase 2 Design

Status: Draft — 2026-05-03 · Owner: J · Drafted by: working session 2026-05-03 Companion to: AUDIT_TRAIL_PRD.md, AUDIT_PHASE_1_DISCOVERY.md, AUDIT_PHASE_1_5_BIPA_AND_OUTCOMES.md

Why this exists. Phase 1 + 1.5 confirmed that today's substrate has no separation between candidate_id (the canonical token) and PII (name, email, phone, address). Both live in workers_500k.parquet. There is no per-access audit. There is no consent gate. There is no retention enforcement. This document specifies the new identity service that will hold the candidate_id ↔ PII mapping, gate every PII read, audit every access, and serve as the single legal-attestable boundary between PII and the rest of the system.

Confirmed by J 2026-05-03: separate daemon (option A in §10.1), signed JSON with PDF render for legal export, legal-only auth credential separate from admin token.

---

1. Scope and non-goals

In scope

• Single source of truth for candidate_id ↔ PII mapping
• Per-PII-access audit log (who/what/when/why)
• Consent + retention metadata (BIPA + Day and Temporary Labor Services Act + healthcare PHI)
• Legal-only access credential, separate from admin tokens
• Healthcare-vertical detection at gateway boundary (per J 2026-05-03 answer 10)
• EU-compatible interface (placeholder fields, lawful-basis tracking, SCC-ready slots — but NOT enforced this phase per J)
• Training-safe export interface (per J 2026-05-03 answer 11)
• Signed-JSON audit response with PDF render path

Out of scope (for this phase)

• The /audit/subject/{id} endpoint itself (Phase 3)
• Subject-tagging across other substrates (Phase 4)
• Right-to-be-forgotten implementation (Phase 7)
• BIPA pre-launch gates — those are Phase 1.6, ahead of this phase

---

2. Architectural shape

2.1 — Process model: separate daemon

Per J's confirmation (2026-05-03), the identity service runs as its own daemon, port-isolated from the gateway. Rationale:

• Single attestable boundary for legal/audit. "All PII access flows through identityd. Show me the identityd access log" is one query, one daemon.
• Independent restart — a gateway crash doesn't take down identity, and an identity panic doesn't break unrelated reads.
• Distinct credential surface — identityd's auth model is wholly separate from gateway's. The legal-only credential exists only in identityd, not in the gateway's JWT issuer.
• Cross-runtime parity — both Rust and Go gateway call identityd over HTTP. There is ONE identity implementation.

Property	Value
Name	identityd
Port	:3225 (Rust legacy line — picks a port adjacent to validatord :3221) and :4225 (Go line)
Implementation language	Go — single implementation, both runtimes call it via HTTP. Avoids re-implementing the audit-log writer + retention sweeper twice.
Storage	Postgres (separate database from any other lakehouse storage). Deployed alongside Langfuse's Postgres or its own; either way, isolated schema with its own grants.
Encryption	Per-row symmetric encryption (AES-256-GCM) of PII columns. Master key in a vault (HashiCorp Vault, AWS KMS, or a sealed-secret file at /etc/lakehouse/identityd_master.key for now). Keys are NEVER logged.
Backup	Standard Postgres backup; keys backed up separately to different storage tier (the cryptographic-erasure model in AUDIT_TRAIL_PRD.md §6 only works if the encrypted-blob backup and the key-backup are not co-located).

2.2 — Schema (Postgres DDL sketch)

-- Single source of truth for the candidate_id ↔ PII mapping
-- Every PII column is stored as ciphertext; keys per row enable
-- per-subject crypto-erasure for RTBF.
CREATE TABLE subjects (
    candidate_id    TEXT PRIMARY KEY,           -- canonical token, e.g. "CAND-000001"
    -- Encrypted PII fields. Each is AES-256-GCM with subject_key_id below.
    -- Plaintext is NEVER stored. NULL means "not collected" not "absent."
    name_ct         BYTEA,
    email_ct        BYTEA,
    phone_ct        BYTEA,
    address_ct      BYTEA,
    ssn_ct          BYTEA,
    dob_ct          BYTEA,
    -- Per-subject encryption key id. Crypto-erasure path: destroy this key
    -- and the ciphertext is unrecoverable, even with the master key.
    subject_key_id  TEXT NOT NULL,
    -- Lawful basis + consent metadata
    consent_status            TEXT NOT NULL,    -- 'pending' | 'given' | 'withdrawn' | 'expired'
    consent_version           TEXT,             -- references published consent template version
    consent_given_at          TIMESTAMPTZ,
    consent_withdrawn_at      TIMESTAMPTZ,
    -- BIPA-specific fields (per Phase 1.5 §1E)
    biometric_consent_status  TEXT,             -- separate from general PII consent
    biometric_retention_until TIMESTAMPTZ,      -- BIPA: max 3 years from last interaction
    -- Vertical detection — drives healthcare PHI routing (per J answer 10)
    vertical                  TEXT,             -- 'general' | 'healthcare' | 'finance' | 'other'
    -- EU-placeholder fields (per J answer 9 — present, not enforced)
    eu_resident       BOOLEAN DEFAULT FALSE,
    lawful_basis      TEXT,                     -- GDPR Art. 6 basis if eu_resident=true
    transfer_mechanism TEXT,                    -- SCC, DPF, BCR — populated when EU comes online
    -- Standard audit columns
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    last_interaction  TIMESTAMPTZ,              -- drives retention sweep
    -- RTBF state
    erased_at         TIMESTAMPTZ,              -- set when crypto-erasure executed
    erasure_reason    TEXT                      -- 'rtbf_request' | 'retention_expired' | 'consent_withdrawn'
);

-- Append-only access audit. EVERY PII read writes a row here.
CREATE TABLE pii_access_log (
    access_id       BIGSERIAL PRIMARY KEY,
    candidate_id    TEXT NOT NULL,
    accessed_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    accessor_kind   TEXT NOT NULL,              -- 'gateway_lookup' | 'audit_response' | 'legal_request' | 'system_resolve'
    accessor_id     TEXT NOT NULL,              -- daemon name + caller token-hash, never raw token
    purpose         TEXT NOT NULL,              -- 'fill_validation' | 'audit_subject_response' | 'admin' | 'legal_audit_DDDDDD'
    fields_accessed TEXT[] NOT NULL,            -- ['name', 'email'] etc.
    request_trace_id TEXT,                      -- ties to Langfuse trace + sessions.jsonl
    integrity_hash  TEXT NOT NULL               -- chain hash for tamper-evidence (this row's hash includes prev row's hash)
);

-- Cryptographic chain for the access log — Merkle-style. Per kimi
-- single-reviewer flag: chain of custody under FRE 901.
-- Each row's integrity_hash = SHA256(prev_hash || row_payload).
-- Last hash periodically committed to a tamper-evident store
-- (could be a separate append-only file with timestamp signing).

-- Per-subject keys table — crypto-erasure target.
-- Destroying a row here makes the corresponding subjects.*_ct unreadable.
CREATE TABLE subject_keys (
    subject_key_id  TEXT PRIMARY KEY,
    candidate_id    TEXT NOT NULL,
    key_material    BYTEA NOT NULL,             -- AES-256 key, encrypted under master key
    created_at      TIMESTAMPTZ NOT NULL,
    destroyed_at    TIMESTAMPTZ,                -- crypto-erasure marker
    destroyed_reason TEXT
);

-- Consent template versioning — BIPA + GDPR + CCPA compliance evidence
CREATE TABLE consent_versions (
    version         TEXT PRIMARY KEY,
    effective_at    TIMESTAMPTZ NOT NULL,
    superseded_at   TIMESTAMPTZ,
    template_text   TEXT NOT NULL,
    biometric_section TEXT,                    -- BIPA-specific clause
    healthcare_section TEXT,                   -- HIPAA-specific clause
    eu_section      TEXT                       -- GDPR-specific clause (placeholder)
);

2.3 — HTTP surface

Identityd exposes a small HTTP surface, all under /v1/identity/:

Method + Path	Purpose	Auth
POST /v1/identity/subjects	Create a new subject. Body: PII fields. Returns: candidate_id (server-generated, NOT sequential — UUID v7 to avoid the kimi enumeration risk).	Gateway/admin token
GET /v1/identity/subjects/{candidate_id}	Resolve PII for a candidate. Returns: requested fields only. EVERY call writes a pii_access_log row.	Service-tier auth — gateway can call but body indicates accessor purpose
GET /v1/identity/subjects/{candidate_id}/full	Return the complete subject record including consent + retention metadata + audit summary. Legal-only credential. Used by audit-response endpoint.	Legal-only token — separate credential, separate rotation
POST /v1/identity/subjects/{candidate_id}/consent	Record consent given/withdrawn, with version.	Gateway/admin token
POST /v1/identity/subjects/{candidate_id}/erase	Crypto-erasure: destroy the subject_key_id, mark erased_at. Idempotent.	Legal-only token
GET /v1/identity/access_log/{candidate_id}	Return the per-subject access log for audit response.	Legal-only token
POST /v1/identity/training_safe_export	Returns identifier-stripped + name-redacted projection of subjects suitable for RAG/training. Logs a "system_resolve" access log row marking export.	Admin token + explicit purpose flag
GET /v1/identity/health	Liveness	None

2.4 — Auth model: legal-only credential

The legal-only credential is materially different from the admin/gateway token:

• Stored at /etc/lakehouse/identityd_legal.token (mode 0400, owner-only)
• Loaded by identityd at startup via systemd EnvironmentFile
• Never logged, never returned in any API response, never crossed with gateway tokens
• Rotation: separate runbook. Triggered by counsel request OR scheduled annually.
• The token's existence is documented in the privacy policy ("legal access requires a separate operator-issued credential, audited per access").

Service-tier auth (gateway-issued) and legal-tier auth (operator-issued) are orthogonal — a request must present the legal token to hit /full, /erase, or /access_log. Even an admin token does not unlock those.

---

3. Integration with the rest of the substrate

3.1 — Gateway changes

When the gateway needs PII for a fill scenario (today this happens by SQL JOIN on candidates/workers_500k), the new flow is:

1. Gateway has only candidate_id (post-§2 view-routing fix from AUDIT_PHASE_1_DISCOVERY §9)
2. Gateway calls GET /v1/identity/subjects/{candidate_id} with purpose=fill_validation and fields=[name] (or [name,phone] etc.)
3. Identityd writes pii_access_log row, decrypts the requested fields, returns them
4. Gateway uses the fields for the validator + tool result, then immediately drops them from in-memory storage after the request completes (no caching)

Critical: the LRU embed cache (commit 150cc3b) currently keys by (model, text) where text contains PII. Post-identity-service, the cache keying must change to (model, candidate_id, field_subset_hash) so the cache key itself is not PII-bearing. This is a Phase 4 task tracked separately.

3.2 — Rust legacy + Go rewrite both call the same identityd

Both gateways (Rust :3100 and Go :4110) call identityd over HTTP. Same endpoints, same auth model. New cross-runtime parity probe audit_parity.sh validates that an identical PII request through both gateways produces identical identityd access-log rows (modulo daemon-name field).

3.3 — outcomes.jsonl + sessions.jsonl writer changes

Per AUDIT_PHASE_1_DISCOVERY §10/C5 (subject_id top-level promotion):

• Change outcomes.jsonl writer to:

  • Add top-level subject_ids: ["CAND-000001", "CAND-000456"] field listing every candidate referenced
  • Strip name from fills[*] rows; replace with name_ref: "[REDACTED-{candidate_id}]" token
  • Authorized callers dereference the token via identityd

• Same for sessions.jsonl SessionRecord: add subject_ids top-level field.

• Same for overseer_corrections.jsonl.

• Same for observerd ops.jsonl when written.

This makes every JSONL sink subject-queryable by subject_id directly, without grepping natural language.

3.4 — Langfuse boundary redaction (per scrum priority C2)

Before the gateway POSTs a chat trace to Langfuse, identityd is consulted to map any PII-shaped substrings in the message array back to candidate_id tokens. Implementation:

• Gateway maintains a per-request map of subject_id → temporarily_resolved_PII for the lifetime of one request
• Before Langfuse POST, gateway iterates message content, replaces resolved PII with [REDACTED-{candidate_id}]
• Langfuse never sees raw names/emails/phones — it sees the tokens, which are unresolvable without a legal-tier identityd call
• For audit, legal counsel can use the token to dereference identity AND see the corresponding Langfuse trace, but Langfuse's storage is PII-free

This addresses the most-dangerous-leak finding from AUDIT_PHASE_1_DISCOVERY §10/C2.

3.5 — Healthcare vertical routing (per J answer 10)

When subjects.vertical = 'healthcare', the gateway routing rules change:

• Tool calls that touch this candidate's data MUST route to local-only models (Ollama on-box), NOT to opencode/openrouter/ollama_cloud egress
• If a healthcare-vertical request can't be served locally, it fails with HTTP 451 ("Unavailable for Legal Reasons") — better to refuse than leak PHI
• The identity service holds the routing decision; the gateway consults it on every call
• Vertical detection itself happens at ingest time (workers_500k row metadata) OR when first PII fetch returns vertical='healthcare'

This requires a one-line addition to the gateway's chat routing in crates/gateway/src/v1/chat.rs + Go-side equivalent in cmd/chatd/main.go. Both should fail-closed: if identityd is unreachable, healthcare requests refuse.

3.6 — Training-safe export (per J answer 11)

POST /v1/identity/training_safe_export returns a projection of subject decision data with:

• name, email, phone, address, ssn, dob ALL stripped (replaced with [REDACTED])
• candidate_id replaced with a hashed pseudonym specific to the export run (different export runs produce different pseudonyms — prevents cross-run correlation)
• Discrimination-proxy phrases (per gemini scrum) detected and [REDACTED-PROXY]-replaced
• Output is suitable for RAG-indexing or fine-tuning corpus building
• An audit log entry documents the export (purpose, requesting accessor, scope, fields)

If a candidate later RTBFs, their pre-export decisions remain in the trained corpus BUT the link back to them is severed (the export pseudonym was random). Legal defense: "the source data was destroyed; the model retains it indistinguishably from synthetic patterns."

---

4. Audit response: what /audit/subject/{id} returns (Phase 3 preview)

Phase 3 builds the audit endpoint, but the shape it returns is dictated by what identityd can produce. Sketch:

{
  "schema": "audit.subject.v1",
  "subject_token": "CAND-000001",
  "request_window": { "from": "2026-01-01", "to": "2026-05-03" },
  "generated_at": "2026-05-03T12:00:00Z",
  "generated_by": "identityd@hostname",
  "integrity_hash": "sha256:...",                       // Merkle-style chain of all decision rows
  "signature": "ed25519:...",                            // identityd signs with its escrow key
  "consent": {
    "status": "given",
    "version": "v3-2026-04-15",
    "given_at": "2026-04-20T14:30:00Z",
    "biometric_consent": "given",
    "biometric_retention_until": "2029-04-20T14:30:00Z"
  },
  "decisions": [
    {
      "ts": "2026-04-22T09:15:23Z",
      "decision_kind": "fill_recommendation",
      "daemon": "gateway",
      "model": "kimi-k2.6",
      "provider": "ollama_cloud",
      "trace_id": "trace-abc",
      "session_id": "session-xyz",
      "input_features": {
        // Sanitized view of what the model saw — no protected attributes,
        // no inferred-attribute proxies, but enough to defend the decision
      },
      "output": "recommended for fill_event_456",
      "rationale": "Skills match: Welder TIG aluminum + 5+ years; geo match: Toledo OH; availability: confirmed",
      "comparator_pool_size": 47,
      "comparator_pool_protected_class_distribution": "see appendix A"  // adverse-impact stats per gemini scrum
    }
  ],
  "comparator_appendix": {
    // EEOC adverse-impact statistics: for the same searches that included
    // this subject, what was the selection rate by protected class?
    // Aggregated; no other subjects' identifiers leak.
  },
  "access_log": [
    // Every PII access for this subject in the request window
    { "at": "...", "purpose": "fill_validation", "fields": ["name"], "trace_id": "..." }
  ],
  "footer": {
    "completeness_attestation": "all decisions about subject_token in the window per retention policy v2 are included",
    "what_was_excluded": "decisions older than 4 years (retention expired) — count: 0",
    "format_version": "audit.subject.v1"
  }
}

PDF render is a downstream consumer — same JSON, different presentation layer (template + signing remains in JSON; PDF is for legal team's final delivery).

---

5. Migration path from current state

This is the single biggest implementation question — how to get from "PII in workers_500k.parquet, no identityd" to "PII in identityd, parquet has only candidate_id + non-PII columns" without breaking the live demo.

Migration strategy: parallel-write, gradual-read-cutover

1. Step 1 — Stand up identityd. Empty database. New service, no callers yet. Health endpoint live. Rust + Go tests can call it but production paths don't.
2. Step 2 — Backfill from workers_500k.parquet. One-shot ETL: read parquet, for each row, write to identityd with consent_status='inferred_existing' (placeholder until counsel writes the real consent backfill story), vertical='general' (correct for non-healthcare data; needs human review for healthcare-flagged rows).
3. Step 3 — Add identityd-call path to gateway behind a feature flag. When LH_USE_IDENTITY_SERVICE=true, the gateway calls identityd for PII; otherwise it uses the legacy SQL path.
4. Step 4 — Cut over reads incrementally. Tool registry first (highest-PII-volume path). Validate via the cross-runtime parity probe audit_parity.sh.
5. Step 5 — Quarantine PII columns in workers_500k.parquet. Once all readers go through identityd, the parquet's PII columns become read-only and eventually moved to a different bucket. The candidate_id-only projection becomes the operational table.

Each step has its own commit, its own gate, and its own rollback. Don't ship steps 2-5 in one commit.

---

6. Cross-runtime parity probe (NEW)

Per AUDIT_PHASE_1_DISCOVERY §4 ask: extend the 5 existing probes with audit_parity.sh. New probe asserts:

1. Same PII fetch through Rust gateway (port 3100) and Go gateway (port 4110) produces identical identityd access-log rows (modulo daemon name)
2. Crypto-erasure of a test subject through Rust gateway is honored when Go gateway tries to fetch
3. Healthcare-vertical routing decision is identical across both runtimes
4. Training-safe export produces byte-identical output regardless of which gateway initiated

Ships as part of Phase 5 (identity service build), not phase 2.

---

7. What this design intentionally does NOT solve

• Does not replace existing protected-attribute exclusion at decision time. The model still sees what the SQL returns; identityd doesn't filter that. Phase 6 of AUDIT_TRAIL_PRD handles boundary enforcement.
• Does not redact pathway memory trace bodies. That's per-trace-write redaction, separate concern. Phase 4.
• Does not retroactively scrub Langfuse history. Past traces still contain PII; only new traces are token-redacted. Counsel may request a one-shot historical-Langfuse purge — that's a separate runbook.
• Does not implement "right to explanation" (GDPR Art. 22 / EU AI Act). Audit response shows decisions; explaining the model's reasoning chain in human-readable form is Phase 8 (legal export format) or its own follow-up phase.
• Does not handle multi-region data residency. Single-region (US-Midwest, by default). EU-placeholder fields are present; multi-region deployment is out of scope.

---

8. Open questions for J before implementation starts

1. Master key location. Vault server, KMS, or a sealed file? Sealed file is fastest to ship; vault is most defensible. Recommend sealed-file for v1 with migration path to vault. Confirm.
2. Postgres for identityd: shared with Langfuse, or its own? Recommend its own — operational isolation. Confirm.
3. vertical field initial values. Backfill all existing subjects to 'general'? Or block backfill until each candidate's vertical is determined? Recommend backfill-to-general + flagging procedure for unknown.
4. Legal-only token issuance procedure. Who has the authority to mint a legal token? Operator (J)? Outside counsel? Both? Recommend J + named outside counsel, dual-control.
5. Crypto-erasure timeline for retention. Default sweep cadence: daily? Weekly? Recommend daily.
6. EU placeholder enforcement timeline. Build the fields now; when do we turn on enforcement? Recommend "when first EU candidate is added; until then, enforcement is no-op."

---

9. Estimated implementation cost

Sub-phase	Effort	Notes
2A — Postgres schema + migrations	4-6 hours	Includes encryption helpers + key management glue
2B — identityd HTTP surface (Go)	1-2 days	All endpoints, auth, signing key, tests
2C — Backfill ETL from workers_500k.parquet	1 day	One-shot script + dry-run mode
2D — Gateway integration (Rust + Go, behind feature flag)	2 days	Per-tool migration, parity probe
2E — outcomes/sessions/observer JSONL writer changes	1 day	Subject_id top-level promotion across all sinks
2F — Langfuse redaction layer	1-2 days	Per-request resolved-PII map + token replacement
2G — Healthcare-vertical routing	0.5 day	Single conditional per gateway
2H — Training-safe export	1 day	The exporter + audit logging
2I — Cross-runtime parity probe audit_parity.sh	0.5 day	New probe, lands in golangLAKEHOUSE
Total	~8-10 working days	Sequential; some can parallelize

This is the largest single phase in the audit-trail program. It's the substrate for everything downstream. Recommend doing it carefully — no half-shipped commits, each sub-phase has its own exit criterion.

---

Change log

• 2026-05-03 — Initial Phase 2 design draft. Incorporates J's confirmed answers (separate daemon, signed JSON+PDF, legal-only auth) plus all Phase 1 + 1.5 findings + scrum-driven priority changes.