lakehouse/docs/AUDIT_TRAIL_PRD.md

# PRD: Production-Ready Audit Trail

**Status:** Draft — 2026-05-03 · **Owner:** J · **Drafted by:** working session 2026-05-03

> **Why this document exists.** Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation.

---

## 1. The legal use case (worked example)

**Scenario.** John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2.

**What we must produce.** A response that proves either:
- (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR
- (b) The system surfaces *exactly* what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI."

**What we must NOT produce.**
- Other subjects' data (response leaks if even one other candidate's name appears)
- Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped)
- Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't *use* them, their presence in the audit log creates new evidence

**The defensibility chain.** The audit shows:
1. **Indexing-time decisions** — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into
2. **Search-time decisions** — every query that included him in candidate sets, what rank he received, what the model used to compute that rank
3. **Recommendation-time decisions** — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned
4. **Iteration decisions** — any iterate retries that touched him (validator failures, model self-corrections)
5. **Outcome decisions** — final fills, rejections, hand-offs

For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale.

---

## 2. The subject audit response — output format

```
GET /audit/subject/{candidate_id}?from=D1&to=D2
→ JSON or signed PDF (legal preference TBD)
```

**Header section:**
- subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash
- pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls.

**Per-decision row schema** (shape, not exhaustive):
```json
{
  "ts": "ISO-8601 UTC",
  "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal",
  "daemon": "gateway | validatord | observerd | matrixd | ingestd",
  "model": "kimi-k2.6 | deepseek-v3.2 | ...",
  "provider": "ollama_cloud | opencode | openrouter",
  "input_features": { /* what the model SAW — sanitized per §4 */ },
  "output": { /* what the model decided */ },
  "rationale": "model's natural-language explanation, or rule-based justification",
  "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree",
  "session_id": "iterate session that produced this row"
}
```

**Footer section:**
- Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy"
- Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited)

---

## 3. Surface map — where decisions happen

| Decision happens at | Currently logged where | Audit-completeness gap |
|---|---|---|
| Ingestion (candidate added to pool) | `data/_kb/outcomes.jsonl`? `journald` mutation log? | UNKNOWN — needs walk |
| Embedding creation (vector built for candidate) | NOWHERE per-candidate; embed cache hits aren't subject-tagged | **MAJOR GAP** — need to subject-tag every embedding |
| Search inclusion (candidate appeared in a result set) | Pathway memory + session JSONL (?) | Partial — need subject-correlation |
| Search rank (position in result set) | Result set in chat traces, but not indexed by candidate | Partial |
| Fill recommendation | `data/datasets/fill_events.parquet` (per CLAUDE.md decision A) + pathway memory | Probably OK but not verified |
| Validation outcome (FillValidator/EmailValidator pass/fail) | `/v1/iterate` session JSONL — but `validation_kind` not populated per yesterday's misread | Partial — fix today |
| Iterate retry escalations | Session JSONL `attempts[]` array | OK |
| Observer signals | observerd events at :3800 (or :4219 Go side) | UNKNOWN — needs walk |
| Matrix-indexer compounding (semantic flags, bug fingerprints) | `pathway_memory/state.json` (currently 91 traces) | Probably leaks — these are tagged by file/task, not by subject |

**Substantive finding from this walk:** the matrix indexer + pathway memory are tagged by *code* not by *subject*. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly.

---

## 4. PII handling rules

**Tokenization rule:** candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces.

**Identity service** (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it.

**Protected-attribute exclusion at decision time:** the model NEVER receives:
- Race, ethnicity, national origin
- Sex, gender, marital status, pregnancy
- Age, date of birth (allowed: years of experience)
- Religion
- Disability, genetic information
- Veteran status (unless legally relevant for the role)
- Sexual orientation, gender identity

If the model never *sees* these, no decision can be predicated on them. The audit row's `input_features` field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input.

**Inferred-attribute risk.** A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. **Open question:** do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls.

**Audit response sanitization:** the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows.

---

## 5. Identity service — candidate_id ↔ PII mapping

**Current state:** `data/datasets/workers_500k.parquet` has the full PII (per CLAUDE.md). The `candidates_safe` view (post-fix `c3c9c21`) is the masked projection. **GAP:** `candidate_id` is currently the row position / a derived field — there's no separate identity service. This needs to change.

**Target state:**
- `identity/` subsystem (new) — holds the `candidate_id → {email, name, address, phone, SSN_last4, DOB, ...}` mapping
- All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see `candidate_id`
- Identity reads require a separate auth credential held by legal-authorized operators
- Every identity read is itself audited (log who accessed PII for which candidate when)
- Identity service runs as its own daemon, port-isolated from the gateway
- Cross-runtime: same identity service backs both Rust and Go

**Open question:** does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary.

---

## 6. Retention policy

**Current state:** UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA.

**Target state (proposed):**
- **Active retention:** while client is in the system, all audit rows kept hot (queryable in <1s)
- **Legal hold:** N years after client/candidate leaves the system, audit rows retained on warm storage. **N is TBD** — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract.
- **Right to be forgotten:** if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed.
- **Cryptographic erasure for append-only logs:** pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable.

**Open question:** does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off.

---

## 7. Current state vs target state

| Capability | Today | Production-ready target | Gap |
|---|---|---|---|
| candidate_id as canonical token | partial (row position?) | UUID, separate from PII | Real — needs identity service |
| Identity service | none | separate daemon, audited reads | Real — build new |
| `/audit/subject/{id}` endpoint | none | live with the §2 schema | Real — build new |
| Subject-tagged embeddings | no | every embed creates an audit row | Real — instrument |
| Subject-tagged search results | partial | every result set logged with subject IDs | Partial — needs walk |
| Subject-tagged validation outcomes | yes (in session JSONL) | yes + integrity-signed | Partial |
| Subject-tagged matrix indexer entries | NO | yes (decide first whether matrix should be subject-aware at all) | Major |
| Protected-attribute filter at decision time | informal | enforced at gateway boundary, audited | Unknown — needs code walk |
| Retention policy | none documented | 4-year hot, configurable cold tier | Real — design + build |
| Right to be forgotten | none | per-subject cryptographic erasure | Real — design + build |
| Cross-runtime parity for all of the above | partial (5 algorithm probes) | new audit-parity probes | Real — extend probe set |

---

## 8. Implementation phases (proposed sequence)

Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds.

### Phase 1 — Discovery walk (read-only, ~3-4 hours)
Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. **No code changes.** Fills in all "UNKNOWN" entries in §3 and §7 with file:line references.

**Exit:** §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells.

### Phase 2 — Identity service design (design doc, ~2 hours)
Write `docs/IDENTITY_SERVICE.md`: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. **No code changes.**

**Exit:** J approves the design.

### Phase 3 — Audit response endpoint (skeleton, ~4-6 hours)
Build `/audit/subject/{id}` endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. **This is the "what John Martinez would get today" baseline.** Reading the output reveals exactly what's wrong.

**Exit:** endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued.

### Phase 4 — Subject tagging across substrates
Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror.

**Exit:** `/audit/subject/{id}` response is *complete* for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end).

### Phase 5 — Identity service build
Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of `workers_500k.parquet` into the new service. Audit every read. Update all callers to never see PII directly.

**Exit:** PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: `audit_parity.sh` validates Rust + Go produce identical audit responses for the same subject.

### Phase 6 — Protected-attribute boundary enforcement
Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's `input_features` becomes load-bearing.

**Exit:** can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping.

### Phase 7 — Retention + right-to-be-forgotten
Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint.

**Exit:** test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows.

### Phase 8 — Legal-format export + signing
Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow.

**Exit:** can produce the John Martinez audit response in the format legal will accept; signature verifies.

### Phase 9 — End-to-end discrimination defense rehearsal
Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them.

**Exit:** J + (eventually) the staffing client's legal team sign off on the format and completeness.

---

## 9. Cross-runtime requirement

**Both Rust legacy and Go rewrite must satisfy every phase's exit criterion.** The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe `audit_parity.sh` lands as part of phase 5.

The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version).

---

## 10. Open questions blocking phase 1

These are the things I need J to decide before phase 1 can start, OR I need to investigate-and-propose:

1. **Identity service: separate daemon vs in-process?** Recommend separate. Confirm.
2. **Retention period N years?** Recommend 4. Need staffing client's legal call.
3. **Photo / surname / zip-code policy?** These are inferred-attribute risks. Need policy decision.
4. **JSON or signed PDF for legal export?** Different downstream costs.
5. **Right-to-be-forgotten under append-only logs:** cryptographic erasure (proposed) or hard delete (breaks integrity)? Confirm crypto-erasure approach.
6. **Audit endpoint auth model:** legal-only credential, or shared with admin? Recommend legal-only with separate token rotation.
7. **The "indexed before search" concern:** matrix indexer + pathway memory currently fingerprint by code, not subject. Do we (a) make them subject-aware (more audit completeness, more PII surface area), (b) keep them code-only and assert in audit response that "no subject-specific compounding state was used," or (c) something else?

Items 1-6 can be resolved by J's call. Item 7 needs design discussion — the safest answer for legal defense is (b), but it loses the "pathway learns about THIS candidate" signal that may be load-bearing for the staffing UX.

---

## 11. What this PRD is NOT

- Not a contract with the staffing client. That document needs lawyers and signs after this is built.
- Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC — passing actual certification is its own project.
- Not a guarantee against discrimination claims. It's a guarantee that *if* a claim is filed, we can produce evidence about how decisions were made.
- Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires.

---

## 12. Appendix — terms

- **Subject** — a person whose data flows through the system (candidate, worker, applicant). Identified by `candidate_id`.
- **Decision** — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.).
- **Audit row** — one record in the audit response per decision, with the schema in §2.
- **PII** — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes.
- **Protected attribute** — characteristics that are illegal to discriminate on under federal/state law. The §4 list.
- **Inferred attribute** — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation).
- **Identity service** — the daemon that holds candidate_id ↔ PII mapping. Separate auth.
- **Subject tagging** — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it.
- **Cryptographic erasure** — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs.

---

## Change log

- 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker.