lakehouse/docs/AUDIT_TRAIL_PRD.md

PRD: Production-Ready Audit Trail

Status: Draft — 2026-05-03 · Owner: J · Drafted by: working session 2026-05-03

Why this document exists. Staffing client won't sign until we can prove the AI system can defend a discrimination claim. We've been claiming "production-ready" off smoke + parity tests; those prove the surface compiles, NOT that an audit response can be produced for a specific person. This PRD writes the audit-trail capability down before we start building it, so the phases are accountable and the scope doesn't drift mid-implementation.

---

1. The legal use case (worked example)

Scenario. John Martinez worked Warehouse B as a placed candidate. Six months later he files a complaint claiming discrimination during the hiring process. His lawyer requests an audit under EEOC discovery: produce every AI-system decision affecting John between dates D1 and D2.

What we must produce. A response that proves either:

• (a) John was treated identically to other candidates with comparable qualifications — same scoring criteria, same model invocations, same decision rules — and the outcome differences are explained by non-protected factors, OR
• (b) The system surfaces exactly what factors led to outcomes, in a form a court can verify, so the claim can be defended on documented criteria rather than "trust the AI."

What we must NOT produce.

• Other subjects' data (response leaks if even one other candidate's name appears)
• Internal infrastructure details (DB paths, server names, internal IDs that aren't candidate-shaped)
• Raw model prompts/completions that contain protected attributes (race, gender, age, etc.) — even if the model didn't use them, their presence in the audit log creates new evidence

The defensibility chain. The audit shows:

1. Indexing-time decisions — when John was added to the candidate pool, what embedding the model produced, what features were extracted, what categories he was placed into
2. Search-time decisions — every query that included him in candidate sets, what rank he received, what the model used to compute that rank
3. Recommendation-time decisions — every fill/recommendation event involving him, what scoring drove it, what validators ran, what they returned
4. Iteration decisions — any iterate retries that touched him (validator failures, model self-corrections)
5. Outcome decisions — final fills, rejections, hand-offs

For each, the audit row must show: timestamp, decision type, model + provider, input features (sanitized of protected attributes — see §4), output decision, rationale.

---

2. The subject audit response — output format

GET /audit/subject/{candidate_id}?from=D1&to=D2
→ JSON or signed PDF (legal preference TBD)

Header section:

• subject identifier (candidate_id), date range, response generation timestamp, signing daemon, integrity hash
• pre-translation note: candidate_id ↔ PII mapping is held by the identity service (§5), NOT by this audit endpoint. Legal counsel re-correlates separately under their own access controls.

Per-decision row schema (shape, not exhaustive):

{
  "ts": "ISO-8601 UTC",
  "decision_kind": "embedding_create | search_inclusion | search_rank | fill_recommendation | validation_outcome | iterate_attempt | observer_signal",
  "daemon": "gateway | validatord | observerd | matrixd | ingestd",
  "model": "kimi-k2.6 | deepseek-v3.2 | ...",
  "provider": "ollama_cloud | opencode | openrouter",
  "input_features": { /* what the model SAW — sanitized per §4 */ },
  "output": { /* what the model decided */ },
  "rationale": "model's natural-language explanation, or rule-based justification",
  "trace_id": "X-Lakehouse-Trace-Id linking to Langfuse trace tree",
  "session_id": "iterate session that produced this row"
}

Footer section:

• Coverage attestation: "this response includes ALL decisions about candidate_id between D1 and D2 that are retained per §6 retention policy"
• Sign-off: cryptographic signature from a daemon whose key is in escrow (proves audit was generated by the system, not hand-edited)

---

3. Surface map — where decisions happen

Decision happens at	Currently logged where	Audit-completeness gap
Ingestion (candidate added to pool)	data/_kb/outcomes.jsonl? journald mutation log?	UNKNOWN — needs walk
Embedding creation (vector built for candidate)	NOWHERE per-candidate; embed cache hits aren't subject-tagged	MAJOR GAP — need to subject-tag every embedding
Search inclusion (candidate appeared in a result set)	Pathway memory + session JSONL (?)	Partial — need subject-correlation
Search rank (position in result set)	Result set in chat traces, but not indexed by candidate	Partial
Fill recommendation	data/datasets/fill_events.parquet (per CLAUDE.md decision A) + pathway memory	Probably OK but not verified
Validation outcome (FillValidator/EmailValidator pass/fail)	/v1/iterate session JSONL — but validation_kind not populated per yesterday's misread	Partial — fix today
Iterate retry escalations	Session JSONL attempts[] array	OK
Observer signals	observerd events at :3800 (or :4219 Go side)	UNKNOWN — needs walk
Matrix-indexer compounding (semantic flags, bug fingerprints)	pathway_memory/state.json (currently 91 traces)	Probably leaks — these are tagged by file/task, not by subject

Substantive finding from this walk: the matrix indexer + pathway memory are tagged by code not by subject. They surface "this code path failed for this task class" — they don't currently let us answer "every decision matrix-indexer made about John." If matrix-indexer fingerprints leak protected-attribute correlations (e.g., a fingerprint that says "candidates from [zip code with majority demographic X] got outcome Y"), that's a discrimination smoking gun that we currently have no way to audit cleanly.

---

4. PII handling rules

Tokenization rule: candidate_id is the only identifier that crosses runtime boundaries (logs, JSONL, traces, pathway memory, observer events, model prompts). Email / name / address / phone / SSN / DOB are NEVER in any of these surfaces.

Identity service (§5) holds the candidate_id ↔ PII mapping. Only legal-authorized access reads it.

Protected-attribute exclusion at decision time: the model NEVER receives:

• Race, ethnicity, national origin
• Sex, gender, marital status, pregnancy
• Age, date of birth (allowed: years of experience)
• Religion
• Disability, genetic information
• Veteran status (unless legally relevant for the role)
• Sexual orientation, gender identity

If the model never sees these, no decision can be predicated on them. The audit row's input_features field proves this: by inspecting the row, a lawyer can confirm protected attributes were absent from input.

Inferred-attribute risk. A model can infer protected attributes from non-protected proxies (zip code → race, name → ethnicity, photo → multiple). The audit must surface this risk. Open question: do we ban photo features from candidate scoring? Do we ban surname tokenization? These are policy calls.

Audit response sanitization: the response goes to the candidate's lawyer, not to the world. It contains the candidate's own name (re-correlated by legal). It must NOT contain other candidates' names, even in comparison/ranking rows.

---

5. Identity service — candidate_id ↔ PII mapping

Current state: data/datasets/workers_500k.parquet has the full PII (per CLAUDE.md). The candidates_safe view (post-fix c3c9c21) is the masked projection. GAP: candidate_id is currently the row position / a derived field — there's no separate identity service. This needs to change.

Target state:

• identity/ subsystem (new) — holds the candidate_id → {email, name, address, phone, SSN_last4, DOB, ...} mapping
• All other systems (gateway, validatord, observerd, matrixd, pathwayd) only ever see candidate_id
• Identity reads require a separate auth credential held by legal-authorized operators
• Every identity read is itself audited (log who accessed PII for which candidate when)
• Identity service runs as its own daemon, port-isolated from the gateway
• Cross-runtime: same identity service backs both Rust and Go

Open question: does the identity service need to be a separate physical daemon (most defensible) or a logically-separated process within an existing one (easier to ship)? Recommend separate daemon — gives legal a single attestable boundary.

---

6. Retention policy

Current state: UNKNOWN. Pathway memory is append-only. Session JSONL is append-only. We have no documented retention SLA.

Target state (proposed):

• Active retention: while client is in the system, all audit rows kept hot (queryable in <1s)
• Legal hold: N years after client/candidate leaves the system, audit rows retained on warm storage. N is TBD — typical EEOC retention is 1-3 years; some state-level claims have 4-year statutes; Title VII discovery can subpoena older. Recommend 4 years minimum, configurable per client contract.
• Right to be forgotten: if a candidate requests deletion under CCPA/GDPR, we apply tombstoning to the identity service (PII removed) BUT preserve the audit-decision rows under candidate_id (anonymized via PII removal at the source). The decision history remains; the human identification is severed.
• Cryptographic erasure for append-only logs: pathway memory and matrix indexer can't be selectively deleted without breaking integrity. Encryption-at-rest with per-subject keys lets us "delete" by destroying the key — the encrypted row remains but is unreadable.

Open question: does the staffing client want a documented retention SLA in their contract? If yes, this PRD becomes contract-grade and the numbers above need their sign-off.

---

7. Current state vs target state

Capability	Today	Production-ready target	Gap
candidate_id as canonical token	partial (row position?)	UUID, separate from PII	Real — needs identity service
Identity service	none	separate daemon, audited reads	Real — build new
/audit/subject/{id} endpoint	none	live with the §2 schema	Real — build new
Subject-tagged embeddings	no	every embed creates an audit row	Real — instrument
Subject-tagged search results	partial	every result set logged with subject IDs	Partial — needs walk
Subject-tagged validation outcomes	yes (in session JSONL)	yes + integrity-signed	Partial
Subject-tagged matrix indexer entries	NO	yes (decide first whether matrix should be subject-aware at all)	Major
Protected-attribute filter at decision time	informal	enforced at gateway boundary, audited	Unknown — needs code walk
Retention policy	none documented	4-year hot, configurable cold tier	Real — design + build
Right to be forgotten	none	per-subject cryptographic erasure	Real — design + build
Cross-runtime parity for all of the above	partial (5 algorithm probes)	new audit-parity probes	Real — extend probe set

---

8. Implementation phases (proposed sequence)

Each phase has an exit criterion the next phase can lean on. Don't start phase N+1 until phase N's exit holds.

Phase 1 — Discovery walk (read-only, ~3-4 hours)

Walk every daemon and tag every code path that touches subject identifiers. Output: a complete map of where candidate_id lives today, where email/name/PII leak today, what's logged where. No code changes. Fills in all "UNKNOWN" entries in §3 and §7 with file:line references.

Exit: §3 surface map is fully populated with current-state evidence. §7 gap table has no "Unknown" cells.

Phase 2 — Identity service design (design doc, ~2 hours)

Write docs/IDENTITY_SERVICE.md: schema, port, auth model, read-audit format, cross-runtime contract, migration path from current state. No code changes.

Exit: J approves the design.

Phase 3 — Audit response endpoint (skeleton, ~4-6 hours)

Build /audit/subject/{id} endpoint that returns ALL information CURRENTLY logged about the subject — even before identity service is built, even if logs leak PII, even if subject-tagging is incomplete. This is the "what John Martinez would get today" baseline. Reading the output reveals exactly what's wrong.

Exit: endpoint returns a JSON response for any candidate_id in workers_500k. Contents are reviewed; gaps catalogued.

Phase 4 — Subject tagging across substrates

Instrument the missing decision points (embedding creation, search rank, observer signals, matrix indexer entries) with subject identifiers. Each daemon's instrumentation lands as its own commit. Cross-runtime: each Rust commit ships with a Go-side mirror.

Exit: /audit/subject/{id} response is complete for the worked example (John Martinez at Warehouse B can be reconstructed end-to-end).

Phase 5 — Identity service build

Stand up the identity daemon. Migrate candidate_id ↔ PII mapping out of workers_500k.parquet into the new service. Audit every read. Update all callers to never see PII directly.

Exit: PII grep across all log files / JSONL streams / pathway memory state returns 0 hits. Cross-runtime parity probe added: audit_parity.sh validates Rust + Go produce identical audit responses for the same subject.

Phase 6 — Protected-attribute boundary enforcement

Add a hard filter at the gateway: any model invocation must declare the input features it sees, and protected attributes are stripped at the boundary. Audit row's input_features becomes load-bearing.

Exit: can run discrimination-test scenario: feed protected attribute through, verify it's stripped before model sees it, verify audit row shows the stripping.

Phase 7 — Retention + right-to-be-forgotten

Document retention SLA. Implement tier-down (hot → warm → cold → encrypted-with-deletable-key). Implement subject-erasure endpoint.

Exit: test scenario: subject requests deletion, identity service tombstones, decision rows remain under candidate_id but are unreadable post-erasure, audit response for that subject returns "subject erased" header instead of decision rows.

Phase 8 — Legal-format export + signing

Decide JSON vs signed PDF for legal output. Build the export pipeline. Sign with a key in escrow.

Exit: can produce the John Martinez audit response in the format legal will accept; signature verifies.

Phase 9 — End-to-end discrimination defense rehearsal

Run the worked example: simulate John Martinez's complaint, generate the audit, walk through what a lawyer would see, identify any remaining gaps, fix them.

Exit: J + (eventually) the staffing client's legal team sign off on the format and completeness.

---

9. Cross-runtime requirement

Both Rust legacy and Go rewrite must satisfy every phase's exit criterion. The 5 existing parity probes (validator, extract_json, session_log, materializer, embed) cover algorithmic equivalence; they do NOT cover audit. New parity probe audit_parity.sh lands as part of phase 5.

The identity service is the new shared substrate — both runtimes call it; the daemon itself is one implementation (no per-runtime version).

---

10. Open questions blocking phase 1

These are the things I need J to decide before phase 1 can start, OR I need to investigate-and-propose:

1. Identity service: separate daemon vs in-process? Recommend separate. Confirm.
2. Retention period N years? Recommend 4. Need staffing client's legal call.
3. Photo / surname / zip-code policy? These are inferred-attribute risks. Need policy decision.
4. JSON or signed PDF for legal export? Different downstream costs.
5. Right-to-be-forgotten under append-only logs: cryptographic erasure (proposed) or hard delete (breaks integrity)? Confirm crypto-erasure approach.
6. Audit endpoint auth model: legal-only credential, or shared with admin? Recommend legal-only with separate token rotation.
7. The "indexed before search" concern: matrix indexer + pathway memory currently fingerprint by code, not subject. Do we (a) make them subject-aware (more audit completeness, more PII surface area), (b) keep them code-only and assert in audit response that "no subject-specific compounding state was used," or (c) something else?

Items 1-6 can be resolved by J's call. Item 7 needs design discussion — the safest answer for legal defense is (b), but it loses the "pathway learns about THIS candidate" signal that may be load-bearing for the staffing UX.

---

11. What this PRD is NOT

• Not a contract with the staffing client. That document needs lawyers and signs after this is built.
• Not a regulatory compliance attestation. We can build to the spirit of GDPR/CCPA/EEOC — passing actual certification is its own project.
• Not a guarantee against discrimination claims. It's a guarantee that if a claim is filed, we can produce evidence about how decisions were made.
• Not a substitute for human review. The audit shows what the AI did; humans still own the final call on hires.

---

12. Appendix — terms

• Subject — a person whose data flows through the system (candidate, worker, applicant). Identified by candidate_id.
• Decision — any system action that changes a subject's standing (added to candidate pool, ranked in search, recommended for fill, validated, scored, etc.).
• Audit row — one record in the audit response per decision, with the schema in §2.
• PII — personally identifiable information per the broad CCPA/GDPR definitions. In this system: name, email, phone, address, SSN, DOB, plus inferred-from-photo attributes.
• Protected attribute — characteristics that are illegal to discriminate on under federal/state law. The §4 list.
• Inferred attribute — a protected attribute the model derives from a non-protected feature (zip → race correlation, name → ethnicity correlation).
• Identity service — the daemon that holds candidate_id ↔ PII mapping. Separate auth.
• Subject tagging — the practice of labeling every decision/embedding/log row with a candidate_id so the audit endpoint can find it.
• Cryptographic erasure — making data unrecoverable by destroying its decryption key, even if the encrypted bytes remain on disk. Used for right-to-be-forgotten on append-only logs.

---

Change log

• 2026-05-03 — Initial draft. Authored after J flagged the audit-trail gap as the production-readiness blocker.