410 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
root
|
68d226c314 |
phase 1.6: BIPA withdrawal endpoint + UI + retention sweep timer
Closes the four production gaps that were live after the consent
endpoint shipped (76cb5ac):
(1) Withdrawal endpoint POST /biometric/subject/{id}/withdraw
backs the BIPA right of withdrawal that consent template v1 §2
explicitly promises. Without it, the only way to honor a
candidate's withdrawal request was the heavier /erase, which
destroys immediately rather than starting the 30-day SLA clock
that the consent template commits to. Side-effects:
- manifest.consent.biometric.status: Given → Withdrawn
- manifest.consent.biometric.retention_until: 18mo → 30d
- audit row kind=biometric_consent_withdrawal, captures
reason + operator_of_record + evidence_path
- DOES NOT touch general_pii or subject.status — biometric
is independently revocable
State machine: Given→Withdrawn (happy), NeverCollected/Pending→
409 nothing_to_withdraw, Withdrawn→409 already_withdrawn (won't
advance the destruction clock), Expired→409 already_expired,
subject Erased/RetentionExpired→403 subject_inactive.
12 new unit tests covering happy path + all guards + a full
grant→withdraw cycle that asserts retention_until is correctly
accelerated and the audit chain has 2 rows in correct order.
(2) Withdraw UI at /biometric/withdraw (mcp-server-served HTML).
3-screen flow: operator auth (token + name in sessionStorage),
withdrawal form (candidate_id + free-text reason ≥10 chars +
optional evidence path), confirmation showing the audit row
HMAC + the 30-day retention_until clock + a curl recipe for
/audit/subject/{id} verification. Same neo-brutalist styling
as biometric_intake.html. Mounted at
http://localhost:3700/biometric/withdraw and externally at
https://devop.live/lakehouse/biometric/withdraw.
(3) Retention sweep systemd timer. crates/catalogd/bin/retention_sweep
binary already existed; this commit schedules it. Daily 03:00 UTC,
Persistent=true so a missed boot triggers on next start. Service
runs as oneshot with --apply (writes a date-stamped JSONL to
data/_catalog/subjects/_retention_sweep_<date>.jsonl ONLY when
overdue subjects exist, per the binary's existing semantics).
install.sh updated to handle .timer + paired .service correctly:
enables the timer, skips direct start of the oneshot service
(the timer pulls it in). One-shot manual test confirmed clean:
100 subjects scanned, 0 overdue (all backfill subjects within
their 4-year general retention window).
(4) operator_of_record bug fix in intake UI. Previously the page
hardcoded the literal string 'intake_ui_operator' as the
operator_of_record sent to /consent — meaning every audit row
captured the same useless placeholder, defeating the whole
point of operator traceability. Fixed by adding an operator
name field to the token-paste step (sessionStorage-backed),
passed through to consent + photo POSTs as the actual operator.
Verified live post-restart:
- gateway /audit/health + /biometric/health both 200
- mcp-server /biometric/intake + /biometric/withdraw both 200
- Live withdraw probes: 401 (no token), 400 (empty body), 404
(ghost subject), 409 nothing_to_withdraw on WORKER-1 (which
is NeverCollected per backfill default) — all expected
- Binary strings contain: process_withdraw, withdraw_consent,
biometric_consent_withdrawal, biometric_withdraw_response.v1,
nothing_to_withdraw, already_withdrawn, already_expired,
/subject/{candidate_id}/withdraw route
- systemd: lakehouse-retention-sweep.timer active+enabled,
next fire Tue 2026-05-05 22:00 CDT (= 03:00 UTC May 6)
- Manual one-shot of retention sweep service: exit 0/SUCCESS,
100 subjects loaded, 0 overdue
83/83 catalogd lib tests + 46/46 biometric_endpoint tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
7f0f500050 |
phase 1.6: candidate intake UI — operator-driven consent + photo capture
Adds the Gate 2 frontend that operators use to capture candidate
biometric consent + photo at intake. Without this, the
/biometric/subject/{id}/consent endpoint shipped in 76cb5ac has
no caller — operators would have to build curl scripts by hand.
Surfaces:
- mcp-server serves /biometric/intake as static HTML (mounted at
http://localhost:3700/biometric/intake; externally accessible via
nginx at https://devop.live/lakehouse/biometric/intake)
- URL must include ?candidate_id=WORKER-XXX
Page flow (4 screens):
- Step 1 — operator authentication. Pastes legal-tier audit token
into a password field; stored in sessionStorage (cleared on tab
close), never localStorage, never cookies.
- Step 2 — consent. Renders the v1 consent template text inline
(Disclosures 1/2/3 + plain-language summary, all matching
docs/policies/consent/biometric_consent_template_v1.md as of
this commit). Captures candidate printed name + checkbox accept;
computes SHA-256 of the rendered consent block as
consent_version_hash; computes SHA-256 of the click-acceptance
evidence (method + name + ts + user_agent + page_origin) and
records it as inline:sha256=<hash> evidence path. POSTs to
gateway /biometric/subject/{id}/consent.
- Step 3 — photo. Two paths: file upload OR getUserMedia camera
capture. Preview before submit. Skip-photo button for
consent-only intake (e.g. consent collected before equipment
available). POSTs to gateway /biometric/subject/{id}/photo.
- Step 4 — confirmation. Displays the audit row HMACs from both
endpoint responses + the verify_biometric_erasure.sh command
the operator can run for chain attestation.
Design choices:
- No framework, no build step. Single self-contained HTML file
~22KB. Matches the existing mcp-server precedent
(onboard.html, console.html, etc.).
- Neo-brutalist dark style matching mcp-server's other pages
(tracked dark surfaces, monospace technical metadata, sharp
borders). Consistent with j's UI preferences.
- Server-authoritative timestamps. The page sends its own UA +
click ts as evidence, but the canonical given_at on the
manifest comes from the gateway's Utc::now() (per
process_consent). Page displays whatever the gateway returns.
- Gateway URL configurable via ?gw= query param; defaults to
http://localhost:3100 for the same-box workstation pattern.
External-access deployment requires a separate nginx route
for the gateway (out of scope for v1 to avoid touching the
devop.live nginx config).
Verified live:
- GET http://localhost:3700/biometric/intake?candidate_id=WORKER-100
returns 200 + 22KB HTML body
- GET https://devop.live/lakehouse/biometric/intake?candidate_id=WORKER-100
returns 200 (nginx /lakehouse/ route proxies to :3700)
- Gateway CORS preflight for POST /biometric/subject/{id}/consent
from origin http://localhost:3700: 200 with allow-methods/headers/origin: *
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
76cb5acb03 |
phase 1.6: add consent-record endpoint POST /biometric/subject/{id}/consent (Gate 2 backend)
Backs the candidate-facing consent flow specified in
docs/policies/consent/biometric_consent_template_v1.md §5. Without
this endpoint, no production code path could flip
consent.biometric.status from NeverCollected/Pending to Given —
meaning every photo upload would fail-closed at the consent gate
(403 consent_required) forever, even after counsel signs the
template. This unblocks the post-cutover intake flow.
Endpoint shape:
- POST /biometric/subject/{candidate_id}/consent
- Auth: legal-tier (X-Lakehouse-Legal-Token header)
- Body (JSON): {consent_version_hash, consent_collection_method,
consent_collection_evidence_path?, operator_of_record}
- consent_collection_method constrained to closed set:
electronic_signature / paper / click_acceptance — operator
typo would silently weaken evidentiary defensibility
- consent_version_hash recorded but not gated against allowlist
at runtime (avoids config-deployment dependency for v2 template
rotation; counsel validates retroactively against
consent_versions table)
State-machine semantics (mirrors upload's double-upload guard):
- NeverCollected / Pending → Given (happy path)
- Given → 409 consent_already_given (re-collect requires explicit
erase + fresh grant under new template version)
- Withdrawn / Expired → 409 consent_post_withdrawal_requires_erase
(explicit erase preserves audit-chain ordering of the cycle)
- Subject status Withdrawn / Erased / RetentionExpired → 403
Server-side authoritative timestamps:
- given_at = Utc::now() (operator-supplied would be tamperable)
- retention_until = given_at + 540 days (18 months per retention
schedule v1 §4). If counsel changes the cap, schedule doc + code
constant CONSENT_RETENTION_DAYS change in the same PR.
Audit row:
- accessor.kind = "biometric_consent_grant"
- accessor.purpose = "version=<hash>;method=<m>;operator=<op>;evidence=<path>"
- fields_accessed = ["consent.biometric.status",
"consent.biometric.retention_until"]
- result = "given"
- Transactional: manifest commit before audit append; rollback
manifest if audit fails.
Tests added (12 new):
- consent_happy_path_flips_status_and_records_audit (full happy
path + audit row inspection + retention_until math)
- consent_missing_token_rejected
- consent_wrong_token_rejected
- consent_subject_not_found_returns_404
- consent_missing_version_hash_400
- consent_missing_method_400
- consent_invalid_method_400
- consent_missing_operator_400
- consent_already_given_returns_409
- consent_post_withdrawal_requires_erase_returns_409
- consent_subject_inactive_returns_403
- consent_grant_then_upload_is_the_intended_intake_flow
(end-to-end: grant → upload → verify chain has 2 rows
consent_grant→biometric_collection in correct order, upload
row chains off the grant's hmac)
71/71 catalogd lib tests + gateway crate compile clean.
Live verification post-restart:
- /audit/health + /biometric/health both 200
- Live POST returns 400 on missing/malformed body, 404 on ghost
subject — auth + body parse + subject lookup ordering verified
- strings(target/release/gateway) contains: record_consent symbol,
biometric_consent_response.v1, biometric_consent_grant,
consent_already_given, consent_post_withdrawal_requires_erase,
electronic_signature, /subject/{candidate_id}/consent route
Production posture: gateway running with the new endpoint live.
The candidate-facing consent UI is NOT yet built; that is a
separate session. This endpoint is the backend the UI will call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
b2c34b80b3 |
phase 1.6: lock Gate 3b = C, reconcile docs to shipped state, fix double-upload file leak
Four threads landing together — all driven by the audit J asked for before
production cutover.
(1) Gate 3b DECIDED: Option C (defer classifications). `BiometricCollection.classifications`
stays `Option<JSON> = None` in v1. `docs/specs/GATE_3B_DEEPFACE_DESIGN.md` status
flipped from "draft / awaits product" to DECIDED. Consent template + retention
schedule revised to remove all "automated facial-classification" / "deepface"
language so disclosed scope matches implemented scope.
(2) Endpoint-path drift reconciled across 3 docs. `PHASE_1_6_BIPA_GATES.md`,
`BIPA_DESTRUCTION_RUNBOOK.md`, and `biometric_retention_schedule_v1.md` had
references to legacy `/v1/identity/subjects/*` paths (proposed under a separate
identityd daemon, never shipped) — corrected to actual shipped routes
`/biometric/subject/*` (catalogd-local). Schema block in PHASE_1_6_BIPA_GATES
rewritten to reflect JSON `SubjectManifest.biometric_collection` substrate
(not the proposed Postgres `subjects` table).
(3) New operational artifacts:
- `scripts/staffing/verify_biometric_erasure.sh` — checks 4 things post-erasure
(manifest cleared, uploads dir empty, audit row matches, chain verified).
Smoke-tested live against WORKER-2.
- `scripts/staffing/biometric_destruction_report.sh` — monthly anonymized
destruction-event aggregation. Smoke-tested clean.
- `scripts/staffing/bundle_counsel_packet.sh` — tarballs the counsel-review
packet with per-file SHA-256 manifest.
- `docs/runbooks/LEGAL_AUDIT_KEY_ROTATION.md` — formal rotation procedure
operationalized after the 2026-05-05 /tmp wipe incident.
- `docs/counsel/COUNSEL_REVIEW_PACKET_2026-05-05.md` — cover note bundling
all eng-staged BIPA docs for counsel review with per-doc questions, sign-off
checklist, recommended review sequence.
(4) Double-upload file leak fixed in `crates/catalogd/src/biometric_endpoint.rs`.
`verify_biometric_erasure.sh` smoked WORKER-2 and surfaced a stranded photo
file. Investigation showed the file was 13-byte test-fixture bytes (zero PII,
no biometric content); audit timeline showed two consecutive uploads followed
by one erasure — the second upload had silently overwritten manifest.data_path,
orphaning the first file. Patched `process_upload` to refuse a second upload
with HTTP 409 + `error: "biometric_already_collected"` when
`biometric_collection.is_some()` on the manifest. Operator must explicitly
POST `/biometric/subject/{id}/erase` first.
Tests: new `second_upload_without_erase_returns_409` (asserts 409 + manifest
pointer unchanged + first file untouched on disk). Replaced
`repeated_uploads_grow_the_chain` with `upload_erase_upload_grows_the_chain_cleanly`
(covers the legitimate re-collection cycle: chain grows to 3 rows). Updated
`content_type_with_parameters_accepted` to use 2 distinct subjects (was
using 1 subject with 2 uploads to test ct parsing — would now 409).
22/22 biometric_endpoint tests + 59/59 catalogd lib tests green post-patch.
Production posture: gateway needs `cargo build --release -p gateway` +
`systemctl restart lakehouse.service` to pick up the new 409 in live traffic.
Counsel calendar is now the only remaining blocker for first real-photo intake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
03e8a91d97 |
STATE_OF_PLAY: 2026-05-05 — audit endpoint recovery + anchor refresh
Reset gateway audit substrate after /tmp wipe disabled it on reboot:
- LH_SUBJECT_AUDIT_KEY moved /tmp/lakehouse_audit/ → /etc/lakehouse/
(canonical persistent path per spec line 112; /tmp wipes on reboot
and silently disabled /audit + /biometric endpoints)
- Fresh 32B HMAC + 44-char legal token at /etc/lakehouse/, mode 0400
- Systemd drop-in updated; gateway restarted; both endpoints 200
- Pre-rotation chains for WORKER-{1..5} (backfill data) will now
tamper-detect under the new key — expected and correct on rotation
Anchor wave-table backfilled with 3 commits that landed after the
last STATE_OF_PLAY refresh on 2026-05-03 evening:
- 7e0112b: retention_sweep stray indent fix
- 848a458: Phase 1.6 Gate 5 erasure endpoint POST /biometric/.../erase
- 8ec43e0: Phase 1.6 Gate 3b deepface integration design doc
Phase 1.6 status table: Gate 5 → eng-DONE; Gate 3b → design-doc-shipped
(recommends Option C defer). Calendar bottleneck text updated.
.gitignore extended for runtime ephemera that surfaced this session:
- data/biometric/ (BIPA-quarantined photos, regulated data)
- reports/scrum/ (local-only review forensics per feedback_audit_findings_log.md)
- experiments/ (per "experiments stay out of tracked tree" policy)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
8ec43e0721 |
phase 1.6 Gate 3b: deepface integration design doc (3 options + recommendation)
Per docs/PHASE_1_6_BIPA_GATES.md Gate 3b. Three viable paths for
populating BiometricCollection.classifications, sized + tradeoff'd:
Option A — Python subprocess per upload (no daemon)
~80 LOC, 0.5-1 day. Smallest integration. Reintroduces a Python
dependency the 2026-05-02 sidecar drop deliberately removed.
Option B — ONNX models in Rust (no Python at all)
~200-400 LOC + model-build pipeline, 5-7 days. Fully consistent
with sidecar drop. Need pre-trained models with appropriate
licenses (or train ourselves, multi-week). Adds face detection
preprocessing in Rust.
Option C — Defer; classifications field stays None
0.25 day. BIPA-safest position; substrate is forward-compatible.
Forces the question "do we actually need classifications?" to be
answered by a real product requirement, not by spec inertia.
Recommendation: **Option C (defer)**, conditional on confirming the
product requirement. Reasoning:
- All BIPA-load-bearing surfaces (consent + audit + retention +
erasure) ship without classifications
- Riskiest BIPA position is collecting demographic-derived data
without a documented business purpose
- Substrate accommodates A or B later in 1-3 days if real demand
surfaces
Open questions for J at the bottom of the doc — pick A/B/C is the
gating decision before any engineering happens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
848a4583da |
phase 1.6 Gate 5: erasure endpoint POST /biometric/subject/{id}/erase
Per docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md. BIPA-defensible
erasure: clears biometric collection (and optionally full PII record),
unlinks the photo file, records the destruction in the per-subject
HMAC chain. The audit row is the legal proof of compliant destruction
even after the underlying data is gone.
Two scopes:
biometric_only (default): clears biometric_collection field, unlinks
the photo, sets consent.biometric.status = withdrawn. Subject
remains active.
full: above PLUS sets status = erased and consent.general_pii.status
= withdrawn. Manifest preserved (proof of destruction); subject
record is logically erased.
Triggers (recorded but not validated against a closed set):
retention_expiry | consent_withdrawal | rtbf | court_order
Body shape:
{
"trigger": "<token>",
"trigger_evidence_path": "<optional path>",
"operator_of_record": "<name>",
"witness": "<name>",
"scope": "biometric_only|full"
}
Response (biometric_erase_response.v1):
candidate_id, scope, trigger, erased_at, fields_cleared,
photo_unlinked, photo_unlink_error, status_after,
biometric_status_after, general_pii_status_after, audit_row_hmac
Order matters for BIPA defensibility:
1. Snapshot original manifest (rollback target)
2. Update manifest (logical erasure)
3. Append audit row (LEGAL proof of intent + scope + operator)
4. Best-effort secure overwrite + unlink photo file (irreversible last)
If audit append fails, manifest is rolled back to original state and
500 returned — the alternative (manifest erased without legal record)
is exactly the silent-failure mode the spec exists to prevent.
If photo unlink fails AFTER audit commits, the response carries
photo_unlinked=false + the error string; operator must manually shred.
Tracing logs the inconsistency loudly.
Tests: 21 unit tests now pass (10 erasure-specific):
- missing token / missing subject / 404
- missing trigger / missing operator / invalid scope (400)
- biometric_only happy path (file unlinked, fields cleared, audit kind=biometric_erasure)
- full scope (status=Erased, general_pii withdrawn, audit kind=full_erasure)
- idempotent on already-erased (audit row records "already_erased" result)
- no-photo case (photo_unlinked=true with no unlink error)
- chain links off prior audit row's row_hmac (NOT GENESIS)
Live verification (post-restart):
- POST /biometric/subject/WORKER-2/erase with consent_withdrawal trigger
→ 200 with all expected fields_cleared + photo_unlinked=true
- Manifest reflects: biometric_collection=null, consent.biometric.status=withdrawn
- GET /audit/subject/WORKER-2: chain_verified=true, 4 rows total,
latest kind=biometric_erasure with operator + trigger in purpose field
- Cross-runtime parity probe: 6/6 byte-identical post-change
Known follow-up (separate bug): photo upload endpoint overwrites
biometric_collection without handling a prior file's data_path —
multiple uploads for the same candidate orphan earlier files. The
erasure endpoint correctly unlinks what the manifest knows about;
operator must shred orphans manually until the upload endpoint
either rejects re-upload (preferred) or maintains a list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
7e0112beb7 |
retention_sweep: fix stray indent on biometric_collection field
Cosmetic — the field was added correctly during the BiometricCollection substrate landing (commit f1fa6e4) but a batch sed left it misindented. Tests still pass (8/8); only the formatting was off. |
||
|
root
|
47a26fdaa8 |
STATE_OF_PLAY: 2026-05-03 evening wave (subject-manifest substrate + Phase 1.6 BIPA)
Documents the 13-commit wave end-to-end: - SUBJECT_MANIFESTS_ON_CATALOGD spec Steps 1-8 SHIPPED - 5/7 Phase 1.6 BIPA gates engineering-complete or eng-staged - 6th cross-runtime parity probe (subject_audit, 6/6 byte-identical) Status table for Phase 1.6 with evidence pointers per item. Operational state captures the LH_SUBJECT_AUDIT_KEY + LH_LEGAL_AUDIT_TOKEN_FILE systemd configuration so next session knows what's in place. Cataloged the three runtime-divergence classes the parity probe loop caught + structurally killed (omitempty stripping, time.RFC3339Nano truncation, json.Marshal HTML-escape). Future Go↔Rust work can reference these patterns instead of rediscovering them. Calendar bottleneck is now counsel review of items 1/2/5/6 — engineering has staged everything it can without legal sign-off. Engineering long pole is Gate 3b (deepface), deferred for design conversation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
3708e6abf1 |
biometric endpoint: scrum-driven hardening
Per 2026-05-03 phase_1_6_gate_3a scrum (10 findings, 0 convergent
location-wise but opus + kimi flagged the same audit-failure issue).
Convergent + load-bearing fix:
Audit-write failure was silently swallowed (returned 200 with empty
hmac) after photo + manifest persisted. For BIPA defensibility this
is wrong — a successful response without an audit row is exactly
the silent-failure mode the spec exists to prevent. Now: full
transactional rollback. If audit append fails after photo + manifest
commit, we remove the photo AND revert the manifest to its
pre-upload state, then return 500 with error="audit_write_failed".
Other real fixes:
Orphan-file leak (opus WARN): if put_subject fails AFTER the photo
is written, the file would orphan on disk with no manifest pointer.
Now removes the photo on manifest-update failure, before returning 500.
Content-Type parameter handling (opus WARN): real-world clients send
`image/jpeg; charset=binary` etc. Parser now strips parameters per
RFC 9110 §8.3 and matches case-insensitively. New regression test
content_type_with_parameters_accepted exercises both.
data_path doc/code mismatch (opus WARN): doc said "relative to the
configured biometric storage root" but code stored absolute path.
Now stores relative — operators reading the manifest reconstruct
the absolute path with their own storage_root, manifests are
portable across deployments. Tests updated.
Timestamp-nanosecond collision (kimi WARN): added 8-char uuid
suffix to filename. Sub-microsecond cadence collision was implausible
but defense-in-depth is cheap.
Dead code (opus + kimi INFO): removed unused require_legal_auth
function (process_upload reimplements the auth check inline)
and the `let _ = ConsentStatus::Given;` no-op type-shape reference.
Skipped (acceptable in v1):
- qwen BLOCK on image format validation: spec explicitly says "we
trust the caller; malformed images fail downstream when deepface
runs in Gate 3b". Documented in the file's module doc-comment.
- qwen WARN on directory create-then-chmod race: brief window
between create_dir_all and set_permissions. Mitigation would
require libc-level umask manipulation; accepted as v1 scope.
- qwen INFO on constant_time_eq duplication: comment explains the
cross-import boundary; acceptable short-term per the reviewer.
Tests: 11 unit tests pass (added content_type_with_parameters_accepted).
Live verification post-restart:
- Content-Type with `; charset=binary` accepted ✓
- data_path returned as relative `WORKER-2/<ts>_<uuid>.jpg` ✓
- Chain verified end-to-end (3 rows: validator + 2 biometric) ✓
- Cross-runtime parity probe still 6/6 byte-identical ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
f1fa6e4e61 |
phase 1.6 Gate 3a: photo upload endpoint with consent gate
Per docs/PHASE_1_6_BIPA_GATES.md §1 Gate 3 (consent-gate substrate).
Deepface classification (Gate 3b) deferred to its own session — needs
Python subprocess design conversation after the 2026-05-02 sidecar drop.
What ships:
shared/types.rs:
- new BiometricCollection sub-struct: data_path, template_hash,
collected_at, consent_version_hash, classifications (Option<JSON>)
- SubjectManifest gains biometric_collection: Option<BiometricCollection>
with #[serde(default)] so existing on-disk manifests parse and
re-emit without drift
catalogd/biometric_endpoint.rs (NEW, ~600 LOC):
POST /subject/{candidate_id}/photo
- Auth: X-Lakehouse-Legal-Token, constant-time-eq compared against
same legal token file as /audit. Same 32-byte minimum.
- Content-Type: must be image/jpeg or image/png (415 otherwise)
- Body: raw image bytes, max 10MB
- 401: missing or wrong token
- 404: subject not registered
- 403: consent.biometric.status != "given" (returns current status)
- 403: subject status in {Withdrawn, Erased, RetentionExpired}
- 200: writes photo to data/biometric/uploads/<sanitized_id>/<ts>.<ext>
with mode 0700 dir + 0600 file, updates SubjectManifest with
BiometricCollection record, appends audit row
(kind="biometric_collection", purpose="photo_upload"), returns
UploadResponse with template_hash + audit_row_hmac.
Logic split: pure async fn process_upload() takes the headers-as-args
so unit tests exercise every branch without HTTP machinery; the
axum handler is just glue. 10 tests covering all 4 reject paths +
happy path + repeated uploads chaining + structural assertion that
the quarantine path is NOT under data/headshots/ (synthetic faces).
gateway/main.rs:
Mounts /biometric on the same condition as /audit — only when the
SubjectAuditWriter is present AND the legal token loads. Storage
root configurable via LH_BIOMETRIC_STORAGE_ROOT (default
./data/biometric/uploads).
Live verification on the running gateway (post-restart):
- GET /biometric/health → "biometric endpoint ready"
- POST without token → 401 auth_failed
- POST with token, no consent → 403 consent_required (status=NeverCollected)
- Flipped WORKER-2 to consent=given, POST → 200 with hash + path
- File at data/biometric/uploads/WORKER-2/<ts>.jpg, mode 0600
- Manifest biometric_collection field reflects the upload
- Audit row chain links cleanly off the prior validator_lookup row
- GET /audit/subject/WORKER-2 returns chain_verified=true, 2 rows
- Cross-runtime parity probe still 6/6 byte-identical post-change
Phase 1.6 status table updated: Gate 3a DONE, Gate 3b (deepface)
deferred. Calendar bottleneck remains counsel review of items 1/2/5/6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
c7aa607ae4 |
phase 1.6 BIPA: scrum-driven fixes
Per 2026-05-03 phase_1_6_bipa_gates scrum (13 findings, 0 convergent).
1 BLOCK verified false positive, 4 real fixes shipped:
False positive (verified):
- opus BLOCK on attest:55 — claimed `set -uo pipefail` without `-e`
makes the post-python3 `if [ $? -ne 0 ]` check unreachable. Verified
WRONG: `X=$(false); echo $?` prints 1. Bash propagates command-
substitution exit through $? on the assignment line. The check IS
the python3 exit gate. Inline comment added to the script noting
the false positive so future scrums don't re-flag.
Real fixes:
1. opus WARN attestation:18 — schema fingerprint hashed names ONLY,
missing column-type changes. A column repurposed to hold base64
photo bytes under its existing name would pass undetected. Now
hashes "name<TAB>type<TAB>nullable=bool" per row. Re-run produced
evidence SHA-256 1fdcc9f1... (vs old 230fffeb..., reflecting the
broader fingerprint scope).
2. opus WARN gate_4_test:60 — definition regex didn't catch
object-literal property forms (`const t = { FEMALE_NAMES: [...] }`)
or TypeScript class fields (`class L { public NAMES_X: string[] = [] }`).
Added two new patterns + a regression test
(Gate 4: object-literal and class-field bypasses are caught) that
exercises 5 bypass forms. 4/4 tests green; 1 minor regex tweak
needed mid-fix to handle single-line class bodies.
3. kimi WARN python3-reliance — script assumed pyarrow installed and
would emit a stack trace into the attestation if not. Added
`python3 -c "import pyarrow"` gate at top with clean install
instructions on failure.
4. opus INFO PHASE_1_6:200 — item 7 (training) silently dropped from
blocking set with bare "deferred" rationale. Now explicitly states
the deferral is conditional on small operator population (J + 1-2
named ops); item 7 re-promotes to blocking if population grows.
⚖ COUNSEL marker added.
Skipped (acceptable as ⚖ COUNSEL placeholders by design):
- kimi WARN consent template:30-day-SLA (counsel decides number)
- kimi WARN consent template:email-placeholder (counsel supplies)
- kimi WARN parquet absence (env override exists; redeployment-aware)
- kimi INFO runbook manual-erasure (marked TODO when /erase ships)
- qwen INFO doc path/status nits (already addressed by file moves)
Tests: 4/4 Gate 4 absence test (incl. new bypass-coverage), 3/3
attestation evidence checks pass on live data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
4708717f6b |
phase 1.6 BIPA gates — engineering wave (4 of 7 staged)
Per docs/PHASE_1_6_BIPA_GATES.md. Status table now reflects:
DONE (engineering-only, no counsel dependency):
- Gate 4: name→ethnicity inference removed from mcp-server.
Removal note in search.html:3372 + new Bun absence test
(mcp-server/phase_1_6_gate_4.test.ts) with 3 assertions:
walker actually scans files, regex catches synthetic positives,
no offending DEFINITION patterns in any .html/.ts/.js source.
3/3 pass.
ENG-DONE, signature pending:
- §2 attestation: scripts/staffing/attest_pre_identityd_biometric_state.sh
runs three checks against the live state:
1. workers_500k.parquet schema has no biometric/photo/face/image col
2. data/_kb/*.jsonl + pathway state contain no base64 image magic
bytes (JPEG /9j/, PNG iVBOR), no data:image/* MIME prefixes,
no field-name patterns ("photo", "biometric", "deepface_*")
3. data/headshots/manifest.jsonl is entirely synthetic-tagged
3/3 evidence checks pass on the live data dir. Generates a
signed-by-operator+counsel attestation document committed at
docs/attestations/BIPA_PRE_IDENTITYD_ATTESTATION_2026-05-03.md
with SHA-256 of the evidence summary so post-signature tampering
is detectable.
ENG-STAGED, awaiting counsel review:
- Gate 1 retention schedule scaffold at
docs/policies/consent/biometric_retention_schedule_v1.md (BIPA
§15(a)). Engineering facts (categories, 18-month operational
ceiling vs 3-year statutory cap, destruction procedure pointer
to Gate 5 runbook) plus ⚖ COUNSEL markers for the binding text.
- Gate 2 consent template scaffold at
docs/policies/consent/biometric_consent_template_v1.md (BIPA
§15(b)(1)-(3)). Required disclosures + plain-language summary +
withdrawal procedure + the structured fields the consent UI must
post to identityd.
- Gate 5 destruction runbook at docs/runbooks/BIPA_DESTRUCTION_RUNBOOK.md.
Triggers, pre-destruction checks (incl. chain-verified gate via
/audit/subject/{id}), procedure (legal-tier endpoint), automatic
audit row append (subject_audit.v1 with kind=biometric_erasure),
backup-window disclosure, monthly reporting cadence, audit-trail
attestation procedure cross-referencing the cross-runtime parity
probe.
BLOCKED on engineering design:
- Gate 3 photo-upload endpoint. Requires identityd photo intake
design + deepface integration scope. Deferred to its own session.
DEFERRED:
- §3 employee training material. Gate 5 runbook §7 may serve as
substrate; counsel decides whether a separate program is needed.
Calendar bottleneck is now counsel review. Engineering can stage no
further deliverables until either (a) Gate 3's design conversation
happens or (b) counsel completes review of items 1/2/5/6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
2222227c16 |
catalogd parity helper: scrum-driven hardening
Per 2026-05-03 step_7_8_retention_and_parity scrum (opus). 5 findings, 0 convergent — but two real fixes shipped: 1. WARN parity_subject_audit.rs:argv — replace .expect() panics with stderr+exit(2). The parity script captures stdout for byte-compare; a Rust panic backtrace lands in stdout (script merges 2>&1) and reads as a parity break instead of a usage error. Added die() helper that mirrors the Go side's error-exit pattern. 2. INFO parity_subject_audit.rs:5 — doc comment hardcoded the absolute path /home/profit/golangLAKEHOUSE/... Replaced with repo-relative reference. INFO findings on retention_sweep argv style + --as-of report path overwrite were noted but not actioned (style only / acceptable for the forecast use case). The major scrum-surfaced bug (Go json.Marshal HTML-escaping <>& while serde_json keeps them literal) is fixed on the Go side in parallel commit. Rust side here is correct as-is — serde_json::to_vec doesn't HTML-escape by default, so no change needed in canonical_json. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
2413c96817 |
catalogd: Step 8 — parity_subject_audit binary (Rust side)
Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 8.
Cross-runtime parity helper consumed by:
golangLAKEHOUSE/scripts/cutover/parity/subject_audit_parity.sh
Two modes:
--known-answer
Print canonical-JSON + HMAC for a hardcoded fixture row. The Go
helper at golangLAKEHOUSE/scripts/cutover/parity/subject_audit_helper/
must produce byte-identical output. Catches algorithm drift
(canonical-JSON sort order, HMAC algorithm, hex encoding).
--verify <audit_log_path> --key <key_path>
Replay the chain on a real production audit log via the live
SubjectAuditWriter::verify_chain (no re-implementation; the actual
production verification path). Output: one JSON line with mode,
count, tip, verified, error.
The helper exercises the SAME verify_chain path the gateway calls, so
algorithm changes in subject_audit.rs automatically flow into the
parity probe.
Live-verified against 5 production audit logs in data/_catalog/subjects;
all 6 parity assertions pass after fixing two real cross-runtime drifts
on the Go side (omitempty trace_id stripping field; time.RFC3339Nano
stripping trailing zero in nanoseconds — both caught by this probe).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
8fc6238dea |
catalogd: Step 7 — daily retention sweep binary
Per docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 7:
"Subjects whose retention.general_pii_until < now AND status != erased
get marked for review (don't auto-delete; legal needs to approve)."
Per shared::types::BiometricConsent doc-comment (BIPA requirement on
biometric data, max 3 years from last interaction):
"Implementation MUST enforce daily expiration sweep against this field."
Therefore the sweep checks BOTH retention clocks. Reports overdue
subjects to data/_catalog/subjects/_retention_sweep_<YYYY-MM-DD>.jsonl.
Idempotent: subjects already in {Erased, RetentionExpired} are skipped
so daily runs do not append duplicate rows.
Does NOT mutate subject manifests. Legal/operator owns the action
(extend, flip status, schedule erasure).
CLI:
retention_sweep # dry-run (default), stderr only
retention_sweep --apply # also write JSONL report
retention_sweep --as-of <RFC3339> # alternate clock for forecast/test
retention_sweep --storage-root <dir> # default ./data
Tests: 8 unit tests on is_overdue covering all 5 SubjectStatus values,
both clocks, BIPA-only path, and idempotency on already-flagged
subjects.
Live verification (100 subjects in ./data/_catalog/subjects):
- now (2026-05-03): 0 overdue (correct — 4-year retention)
- --as-of 2031-06-01: 100 overdue, 394 days past, jsonl report shape
verified with biometric fields correctly omitted via
serde skip_serializing_if when subject has no biometric clock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
2a4b316a15 |
subjects: 2nd scrum fix wave (token min, chain_tip, tampering, rebuild collision warn)
Second cross-lineage scrum on Steps 5+6 returned 13 distinct findings, 0 convergent.
Three BLOCK-class claims verified as false positives (cache IS written, per-subject
Mutex IS in place, spawn IS safe under writer's lock). Five real fixes shipped:
1. audit_endpoint: legal token min length 16->32 (HMAC-SHA256 best practice, kimi)
2. subject_audit: new chain_tip() returns last hash from full log; audit_endpoint
now reports chain_root from full chain instead of windowed slice (opus)
3. registry: rebuild loader now warns on sanitize collision (symmetric with
put_subject's collision guard - opus)
4. audit_endpoint: tampering detection - if manifest expects non-empty chain_root
but log returns 0 rows, flag chain_verified=false with explicit message (opus)
5. execution_loop::audit_result_state: tightened heuristic - error/denied/not_found
only classify when no rows/data/results sibling (opus INFO)
Tests: 17 catalogd subject + 6 gateway audit_result_state, all green.
New: audit_result_state_does_not_classify_error_when_data_sibling_present,
audit_result_state_status_is_authoritative_even_with_data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
15cfd76c04 |
catalogd + gateway: Step 6 — /audit/subject/{id} legal-tier HTTP endpoint
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 6
+ §4 (response shape) + §6 (auth model). The defense-against-EEOC-
discovery surface is live: legal counsel hits one URL with one token,
gets back a signed-by-HMAC-chain audit response naming every PII access
for a subject in a time window.
New module: crates/catalogd/src/audit_endpoint.rs (~340 LOC)
- AuditEndpointState { registry, writer, legal_token }
- router() exposes:
GET /subject/{candidate_id}?from=ISO&to=ISO (full audit response)
GET /health (liveness + token check)
- require_legal_auth() — constant-time-eq compare against the
X-Lakehouse-Legal-Token header. Avoids timing leaks on the token
check without pulling in `subtle` for one comparison.
- Token loaded from /etc/lakehouse/legal_audit.token (env-overridable
via LH_LEGAL_AUDIT_TOKEN_FILE). Empty file or <16 chars = endpoint
serves 503 with a clear reason. Token value NEVER logged.
- Response schema: subject_audit_response.v1 with manifest +
audit_log (rows + chain verification) + datasets_referenced +
safe_views_available + completeness_attestation.
New helper on SubjectAuditWriter:
- read_rows_in_range(candidate_id, from, to) — returns rows in window,
used by the endpoint to assemble the response without re-reading
the entire chain.
- verify_chain() now returns Ok(0) when the audit log file doesn't
exist (empty = trivially valid). Prevents legitimate "no PII access
yet for this subject" from showing as integrity=BROKEN in the
audit response. Caller can detect "log was deleted" via comparison
to SubjectManifest.audit_log_chain_root (when that mirror lands).
main.rs:
- Audit endpoint mounted at /audit ONLY when both subject_audit
writer AND legal token are present. Disabled-by-default keeps the
surface from accidentally serving in dev/bring-up environments
without proper credentials.
Tests (9/9 passing):
- constant_time_eq (correctness on equal/diff/empty/length-mismatch)
- missing_legal_token_returns_503
- missing_header_returns_401
- wrong_token_returns_401
- correct_token_passes_auth
- audit_response_assembly_full_path (manifest + 3 rows + chain verify)
- audit_response_window_filters_rows (time-bounded window)
- empty_token_file_results_in_disabled_endpoint
- short_token_file_rejected_at_load (<16 char min)
LIVE end-to-end verification:
1. Plant signing key + legal token in /tmp/lakehouse_audit/
2. Restart gateway with LH_SUBJECT_AUDIT_KEY + LH_LEGAL_AUDIT_TOKEN_FILE
pointing at the test files
3. /audit/health → 200 "audit endpoint ready"
4. /audit/subject/WORKER-1 (no token) → 401 "missing X-Lakehouse-Legal-Token"
5. /audit/subject/WORKER-1 (wrong token) → 401 "X-Lakehouse-Legal-Token mismatch"
6. /audit/subject/WORKER-1 (correct token) → 200 + full manifest + 0 rows
+ chain_verified=true (empty log path)
7. POST /v1/validate with candidate_id=WORKER-1 → triggers WorkerLookup.find()
via the AuditingWorkerLookup wrapper from Step 5
8. data/_catalog/subjects/WORKER-1.audit.jsonl now exists with 1 row
(accessor.purpose=validator_worker_lookup, result=not_found,
prev_chain_hash=GENESIS, valid HMAC)
9. /audit/subject/WORKER-1 (correct token) → 200 + manifest + 1 row +
chain_verified=true + chain_rows_total=1 + completeness attestation
The full audit-trail loop (PII access → audit row → chain → audit response)
works end-to-end on the live gateway.
NOT in this commit (future steps):
- Step 7: Daily retention sweep
- Step 8: Cross-runtime parity (Go side reads the same shapes)
- Mirror chain root to SubjectManifest.audit_log_chain_root after
each append (so tampering detection can use the manifest's
cached root as ground truth)
- Live row projection from datasets (currently caller follows up
via /query/sql against the safe_views named in the response)
- Ed25519 signature on the response (chain verification IS the v1
attestation; signing is future hardening per spec §10)
cargo build --release clean. cargo test -p catalogd audit_endpoint
9/9 PASS. Live verification successful.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
cd8c59a53d |
gateway: Step 5 — wire SubjectAuditWriter into validator WorkerLookup
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 5.
Every WorkerLookup.find() call from the validator path now produces
one audit row in the per-subject HMAC-chained JSONL. Failures are
non-blocking — validator continues whether audit succeeds or fails.
Approach: decorator pattern. WorkerLookup is a sync trait by design
(validator's contract is "in-memory snapshot, no per-call I/O") and
audit writes are async, so we can't expand the trait. Instead, a
new AuditingWorkerLookup wraps the inner lookup, captures a
tokio::runtime::Handle at construction, and spawns audit writes from
sync find() onto that handle. The chain stays intact under spawn fan-
out because the writer's per-subject Mutex (shipped in the previous
scrum-fix commit) serializes same-subject appends regardless of how
the spawn calls arrive.
Files changed:
crates/gateway/src/v1/auditing_worker_lookup.rs (NEW, 175 LOC):
- AuditingWorkerLookup<inner: dyn WorkerLookup, audit: Option<Arc<Writer>>>
- new() captures Tokio Handle if audit is Some
- find() runs inner lookup, then spawns audit append with:
accessor.kind = "validator_lookup"
accessor.purpose = "validator_worker_lookup"
fields_accessed = ["exists"] (validator only proves existence
of a subject; downstream code reads policy
fields separately and would have its own
audit if those become PII)
result = "success" if found, "not_found" otherwise
- Audit-disabled path (audit: None) is a transparent passthrough
— zero overhead, no panic, no runtime requirement.
crates/gateway/src/v1/mod.rs:
+ pub mod auditing_worker_lookup;
crates/gateway/src/main.rs:
- Hoisted subject_audit_writer construction OUT of the V1State
literal (declaration-order constraint: validate_workers needs
access to the writer). The hoisted Arc is then reused for the
V1State.subject_audit field.
- validate_workers now wraps the raw lookup with
AuditingWorkerLookup::new(raw, subject_audit_writer.clone())
Tests (4/4 passing):
- find_existing_subject_writes_success_audit_row
- find_missing_subject_writes_not_found_audit_row (phantom-id case)
- audit_disabled_means_no_writes_no_overhead (None pathway)
- many_finds_to_same_subject_produce_intact_chain (30 sequential
spawns on the same subject — chain verifies all 30, regression
against the race we fixed in catalogd subject_audit)
Also catches the iterate.rs:324 phantom-ID check transparently —
that codepath calls state.validate_workers.find(...) which now goes
through the wrapper, so every phantom-id rejection logs an audit row
for free.
NOT in this commit (future steps):
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
- Threading X-Lakehouse-Trace-Id from request through to audit row
(currently audit row's accessor.trace_id is empty)
cargo build --release clean. cargo test -p gateway auditing_worker_lookup
4/4 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
e38f3573ff |
subject manifests Steps 1-4 — fix scrum-flagged BLOCKs and WARNs
2026-05-03 cross-lineage scrum on the subjects_steps_1_to_4 wave
returned 14 distinct findings, 0 convergent. opus verdict was HOLD
with 3 BLOCKs around the audit-chain integrity. All real. Fixed:
──────────────────────────────────────────────────────────────────
BLOCK 1 — opus subject_audit.rs:172 + execution_loop.rs:391
Concurrency race: append_line is read-modify-write; the gateway
hook used tokio::spawn fan-out → two concurrent appends to the
same subject both read the same prev_hash, both compute their
HMAC from the same prev, second write silently overwrites first
→ row lost AND chain broken.
Fix:
- SubjectAuditWriter gains per-subject Mutex map. append() acquires
the subject's lock for the duration of the read-modify-write.
Different subjects still parallelize.
- Gateway hook switches from tokio::spawn to inline await. Per-row
cost is ~1ms (one object_store put); inline is correct AND cheap.
- New regression test: 50 concurrent appends to the same subject,
asserts all 50 land with intact chain.
BLOCK 2 — opus subject_audit.rs:108
Non-deterministic canonicalization: serde_json serializes struct
fields in declaration order. Schema evolution (adding/reordering
fields) silently changes the bytes verify_chain hashes → chain
breaks even when nothing was actually tampered with.
Fix:
- New canonical_json() free fn — recursive value rewrite to sort
object keys alphabetically (BTreeMap projection), arrays preserve
order, scalars pass through. Stable across struct evolution.
- Both append() and verify_chain() now compute HMAC over canonical
bytes, not declaration-order bytes.
- New regression tests: alphabetical-key + array-order-preserved.
WARN — opus execution_loop:401
Audit row's `result` was hardcoded to "success" for every Ok(result)
including payloads like {"error":"not found"}. Misleads compliance.
Fix:
- New audit_result_state() free fn that inspects the payload
top-level for error/denied/not_found/status signals (per spec
§3.2 enum). Defaults to "success" only when no error signal.
- 4 new tests covering each enum case + falsy-signals defense.
WARN — opus registry.rs:735
Storage-key collision: sanitize_view_name(id) is the disk key,
but the in-memory HashMap was keyed by raw candidate_id. Two
distinct ids that sanitize to the same key (e.g. "CAND/1" and
"CAND_1") would collide on disk while appearing distinct in
memory; second put silently overwrites first; rebuild loads only
one.
Fix:
- put_subject() / get_subject() / delete_subject() / rebuild()
all key the in-memory HashMap by sanitize_view_name(id), matching
the storage key shape.
- Collision guard: put_subject() refuses (with clear error) when
the sanitized key matches an EXISTING subject with a DIFFERENT
raw candidate_id.
- New regression test: put("CAND/1") then put("CAND_1") errors
+ first subject survives.
WARN — opus backfill_subjects.rs:189
trim_start_matches strips REPEATED prefixes; the spec wanted
one-shot semantics. Edge case unlikely in practice but real.
Fix:
- Switched to strip_prefix(&prefix).unwrap_or(&cid). One-shot.
INFO — opus subject_audit.rs:131
Per-byte format!("{:02x}", b) allocates each iteration. Hot path
on every append.
Fix:
- Replaced with const HEX lookup table + push() into preallocated
String. Same output bytes, no per-byte allocation.
──────────────────────────────────────────────────────────────────
Test summary post-fix:
catalogd subject_audit: 11/11 PASS (added 4 new — concurrency
race regression, parallel-different-subjects,
canonical-key sort, canonical-array order)
catalogd registry subject: 6/6 PASS (added 1 new — collision guard)
gateway execution_loop subject: 10/10 PASS (added 4 new —
audit_result_state enum coverage)
All 27 subject-related tests green. cargo build --release clean.
The convergent-zero scrum result was misleading on its face — opus
caught real BLOCKs that kimi/qwen missed. Per
feedback_cross_lineage_review.md: opus is the load-bearing reviewer;
single-opus BLOCKs warrant manual verification, which here confirmed
all three were correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
fef1efd2ac |
gateway: Step 4 — wire SubjectAuditWriter into tool dispatch
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 4.
Every tool result that returns rows referencing a subject now produces
audit rows in the per-subject HMAC-chained JSONL. Failures non-blocking.
Changes:
v1/mod.rs:
+ V1State.subject_audit: Option<Arc<SubjectAuditWriter>>
(None when key file missing — audit becomes a no-op with warning,
PII paths still serve)
main.rs:
+ Construct SubjectAuditWriter at startup from
LH_SUBJECT_AUDIT_KEY env or /etc/lakehouse/subject_audit.key.
Missing/short key = log warning + leave None (gateway boots, audit
disabled). Same store as the rest of catalogd.
execution_loop/mod.rs:
+ audit_subject_hits_in() — called after every successful tool
dispatch. Walks the result JSON, finds candidate_id / worker_id
fields, fires one SubjectAuditRow per (subject, fields) pair.
Tokio::spawn so audit latency never adds to tool path.
+ collect_subject_hits() — free fn, recursive JSON walker. Handles:
"candidate_id":"X" → audit candidate_id="X"
"worker_id":42 → audit candidate_id="WORKER-42" (matches
backfill convention)
"worker_id":"42" → audit candidate_id="WORKER-42" (string form)
Other fields in the same object become fields_accessed (so audit
row records "this access surfaced name + phone for candidate X").
Ignores objects without id fields. Skips empty id strings. Recurses
through nested objects + arrays.
Tests (6/6 passing — gateway::collect_subject_hits_*):
- finds_candidate_id_strings (basic case + fields_accessed extraction)
- prefixes_worker_id_int (int → WORKER-N)
- handles_worker_id_string (string → WORKER-N)
- recurses_through_nested_objects (joins / mixed payloads)
- ignores_objects_without_id_fields (no false positives)
- skips_empty_id_strings (defensive)
Per spec §3.2: failures are logged, never propagated. Better to leak
an audit row than block a tool response. Operators monitor warning
volume to detect audit-write regressions.
NOT in this commit (future steps):
- Step 5: Wire validator WorkerLookup similarly (each candidate_id
resolved by FillValidator gets an audit row)
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
- Mirror chain root to SubjectManifest.audit_log_chain_root after
each append (currently the chain is verifiable via verify_chain()
even without the manifest mirror; the mirror is an optimization)
- Thread X-Lakehouse-Trace-Id from request through to audit row
cargo build --release clean. cargo test -p gateway collect_subject_hits
6/6 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
bce6dfd1ee |
catalogd: Step 3 — backfill_subjects binary (BIPA-defensible defaults)
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md §5 Step 3.
Reads a parquet source, creates one SubjectManifest per row with the
spec-defined safe defaults, persists via Registry::put_subject().
Defaults baked in (per spec §2 + §5 Step 5):
- vertical = unknown (HIPAA fail-closed)
- consent.general_pii = pending_backfill_review (NOT inferred_existing — BIPA defense)
- consent.biometric = never_collected (no biometric data backfilled)
- retention.general_pii_until = now + 4 years
- retention.policy = "4_year_default"
Conservative ergonomics:
- --limit 1000 by default. --all to do the full source.
- --dry-run for parse + count + sample without writes.
- --concurrency 32 (bounded via tokio::sync::Semaphore).
- Idempotent: skips subjects that already exist in catalog.
- Progress reports every ~5% (or 5K rows, whichever smaller).
Live verification on workers_500k.parquet:
--limit 100 dry-run: parsed 100 rows, sampled WORKER-1..5, 0 writes ✓
--limit 100 commit: 100 inserted, 0 failed, 100 files in
data/_catalog/subjects/ ✓
--limit 100 re-run: 0 inserted, 100 skipped (idempotent) ✓
Sample manifest (data/_catalog/subjects/WORKER-1.json):
{
"schema": "subject_manifest.v1",
"candidate_id": "WORKER-1",
"status": "active",
"vertical": "unknown",
"consent": {
"general_pii": {"status": "pending_backfill_review", ...},
"biometric": {"status": "never_collected", ...}
},
"retention": {"general_pii_until": "2030-05-02T...", "policy": "4_year_default"},
"datasets": [{"name": "workers_500k", "key_column": "worker_id", "key_value": "1"}]
}
NOT in this commit (future steps):
- Step 4: Wire gateway tool registry to write audit rows on every
candidate_id returned (uses SubjectAuditWriter from Step 2)
- Step 5: Wire validator WorkerLookup similarly
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
- Backfill the full 500K (operator decision: --all when ready;
note: 500K JSON files in one dir will slow startup load — may
want SQLite/single-file backend before that scale)
Operator note: backfill is run-once. To extend to candidates table,
re-run with --dataset candidates --key-column candidate_id (no prefix
since candidate_id is already the canonical token there).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
d16131bcab |
catalogd: Step 2 — SubjectAuditWriter with HMAC chain
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 2.
Per-subject append-only audit JSONL with HMAC-SHA256 chain. Local-first
— no Vault, no external anchor (those are v2 if SOC2 Type II becomes
contract-required; v1 deliberately stays small).
shared/types.rs additions:
- AuditAccessor — kind, daemon, purpose, trace_id
- SubjectAuditRow — schema/ts/candidate_id/accessor/fields_accessed/
result/prev_chain_hash/row_hmac
crates/catalogd/src/subject_audit.rs (NEW):
- SubjectAuditWriter — holds signing key + per-subject latest-hash cache
- from_key_file() — loads key from sealed file, requires ≥32 bytes
- with_inline_key() — for tests + bring-up
- append() — computes HMAC chain link, persists JSONL row, returns new
chain root (caller mirrors to SubjectManifest.audit_log_chain_root)
- verify_chain() — full re-verification of a subject's audit log,
catches both prev_hash drift AND row-level HMAC tampering
- scan_latest_hash() — cold-start path, finds prev_hash from JSONL tail
- append_line() — read-modify-write pattern (object stores have no
native append; same shape as the rest of catalogd's persistence)
Crypto: HMAC-SHA256 via the standard `hmac` crate (added to workspace
+ catalogd deps; not implementing crypto by hand). Output is lowercase
hex matching the rest of the codebase's SHA-256 conventions.
Security choices:
- NO Debug impl on SubjectAuditWriter — auto-deriving Debug would risk
leaking the signing key into log lines. Tests work around this by
matching on Result instead of using .unwrap_err().
- Key min length 32 bytes (HMAC-SHA256 block size guidance).
- Failures are NOT swallowed — Result returned, caller decides whether
to log + continue (per spec §3.2 the gateway tool registry SHOULD
log + continue rather than block reads).
Tests (7/7 passing):
- first_append_uses_genesis_prev_hash
- chain_links_each_append (3-row chain verifies)
- separate_subjects_have_independent_chains (per-subject isolation)
- tamper_detected_on_verify (mutation in middle of chain breaks verify)
- cold_writer_picks_up_existing_chain (process restart preserves chain)
- empty_candidate_id_rejected
- key_too_short_rejected_via_file
NOT in this commit (future steps):
- Step 3: Backfill ETL from workers_500k.parquet (next per J)
- Step 4: Wire gateway tool registry to call append() on every
candidate_id returned by search_candidates / get_candidate
- Step 5: Wire validator WorkerLookup similarly
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
- Mirroring chain root to SubjectManifest.audit_log_chain_root
(separate concern; do at the call site)
cargo check --workspace clean. cargo test -p catalogd subject_audit
7/7 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
d25990982c |
catalogd: Step 1 — SubjectManifest type + Registry CRUD
Implementation of docs/specs/SUBJECT_MANIFESTS_ON_CATALOGD.md Step 1.
Mirrors the existing AiView put/get/list/delete pattern. NOT a separate
daemon, NOT new infrastructure — extends catalogd's manifest layer with
a fourth manifest type (subject) alongside dataset/view/tombstone/profile.
shared/types.rs additions:
- SubjectManifest (the wire format from spec §2)
- SubjectStatus enum: pending_consent | active | withdrawn |
retention_expired | erased
- SubjectVertical enum: unknown | general | healthcare | finance | other
(default = Unknown for fail-closed routing per spec §2.1)
- ConsentStatus enum: pending_backfill_review | pending_first_contact |
given | withdrawn | expired
- BiometricConsentStatus enum: never_collected | pending | given |
withdrawn | expired
- GeneralPiiConsent + BiometricConsent + SubjectConsent
- SubjectRetention (general_pii_until + policy)
- SubjectDatasetRef (name + key_column + key_value pointing at existing
catalogd dataset manifests)
catalogd/registry.rs additions:
- subjects: Arc<RwLock<HashMap<String, SubjectManifest>>> field on Registry
- put_subject() — validates dataset refs, persists to
_catalog/subjects/<id>.json, updates in-memory cache
- get_subject() / list_subjects() / delete_subject() / subjects_count()
- rebuild() now loads subject manifests at startup alongside views +
profiles + tombstones
Tests (5/5 passing):
- put_subject_with_no_dataset_refs_succeeds
- put_subject_rejects_dangling_dataset_ref (validation works)
- put_subject_with_valid_dataset_ref_succeeds
- subject_round_trips_through_object_store (persistence works)
- delete_subject_removes_in_memory_and_persistence
NOT in this commit (future steps):
- Step 2: SubjectAuditWriter with HMAC chain
- Step 3: Backfill ETL from workers_500k.parquet
- Steps 4-5: Wire gateway tool registry + validator to write audit rows
- Step 6: /audit/subject/{id} HTTP endpoint
- Step 7: Daily retention sweep
cargo check --workspace clean. cargo test -p catalogd subject 5/5 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
ed1fcd3c26 |
specs: pathway_memory v1 + subject_manifests_on_catalogd v1
Two specifications addressing the framing J asked for after reading
the llms3.com blog: standardize what we have so future work doesn't
drift, and apply the local-first thesis to the audit problem instead
of the over-scoped SaaS-tier identity service.
PATHWAY_MEMORY_SPEC.md (~400 lines):
Documents the existing crates/vectord/src/pathway_memory.rs as a
spec — the third metadata layer alongside catalogd's data metadata
and playbook_memory's operational memory. Defines:
- PathwayTrace wire format
- pathway_id = SHA256(task_class | file_prefix | signal_class)
- file_prefix algorithm (first 2 path segments)
- pathway_vec: 32-bucket bag-of-tokens hash, fixed dim per spec
- Lifecycle: insert → revise → replay → probation gate retire
- Mem0 versioning (trace_uid + parent_trace_uid + version chain)
- Access patterns: query_for_hotswap / query_by_vec / list_versions
- PII risk surface (reducer_summary + final_verdict)
- Spec boundary: stable in v1 vs implementation-specific
No new architecture. Descriptive, not prescriptive.
SUBJECT_MANIFESTS_ON_CATALOGD.md (~400 lines):
The local-first audit-trail spec. Adds a fourth manifest type to
catalogd alongside dataset/view/tombstone/profile. NOT a separate
identity daemon. NOT Vault/KMS/dual-control JWT. Builds on
primitives catalogd already ships:
- SubjectManifest at data/_catalog/subjects/<id>.json
- Per-subject HMAC-chained audit JSONL
- Daily retention sweep using existing tombstone primitives
- Vertical-aware routing (healthcare → local-only)
- Legal-tier credential separate from gateway internal auth
~4 days estimated implementation effort vs 17-20 days for the
IDENTITY_SERVICE_DESIGN approach. Same defensibility for the
staffing-client launch window. Strictly additive to compatibility
with the v3 design if SOC2 Type II becomes a contract requirement.
These are SPECS — what the system already does (pathway) and what's
the smallest local-first thing that addresses the audit need
(subject manifests). Not 9-phase plans. Not new daemons.
The pathway spec is descriptive: writing down what exists so the
next person doesn't reinvent it. The subject-manifests spec is
prescriptive: J greenlights, implementation is days not weeks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
991db7be1a |
scrum: wrapper resilience + Python tz deprecation fix
Two small fixes after testing the tool end-to-end:
1. scrum_review.sh's tally-aggregation step occasionally exits non-zero
even when the per-reviewer markdown verdicts are all written
correctly. Wrapper was bailing on that exit code and dropping the
KB write. Now the wrapper:
- Treats scrum_review.sh's exit code as advisory
- If a tally markdown exists, uses it
- If only per-reviewer markdowns exist, auto-rebuilds a minimal
tally listing them (so KB row still gets written)
- Only bails if NO per-reviewer verdicts at all
2. Python `datetime.utcnow()` deprecated. Switched to
`datetime.now(datetime.UTC).isoformat().replace("+00:00", "Z")`
for the same Z-suffix shape callers expect.
Verified: ./scrum --since=HEAD~1 now writes KB row even when
scrum_review.sh's tally step has issues. KB rows: 2 (and growing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
dbcd05c5c5 |
audit docs: deprecation headers — over-scoped for local-only deployment
Today's PRD-line-70 reframe (everything runs locally) means the audit-trail docs I drafted earlier this session are over-engineered for J's actual deployment model. They were sized for SaaS-tier infra (Vault/KMS/S3 Object Lock/dual-control JWT/separate Postgres) — appropriate for a multi-tenant cloud service, wrong for a single-box local install. Adding clear deprecation headers so future sessions don't read these as authoritative and propose another 17-20 day plan involving cloud infrastructure that would re-violate PRD line 70. What STAYS valid (preserved in headers): - The legal use case (John Martinez worked example) - The IL/IN jurisdictional surface (counsel checklist) - The Phase 1 + 1.5 discovery findings (PII flow paths file:line) - Phase 1.6 BIPA gates (when real photos arrive) What's OVER-SCOPED (flagged in headers): - The 9-phase implementation plan - The identity service design (Vault/KMS/dual-control) Future v2 of these docs needs to be sized for local single-box: a few hundred LOC of local writers + signed local audit file, not 17-20 days of distributed-systems design. No code changes. Just doc-level guardrails for future scope drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
5f40b7a312 |
STATE_OF_PLAY: lock in today's reverts as DO NOT RELITIGATE
Five new entries to prevent today's cleanup from being undone by future sessions or future PRs that don't read the full context: - PRD line 70 load-bearing — local-only on customer hot path. PR #13's cloud-routing defaults reverted (d054c0b). Cloud is opt-in dev-only. - /v1/usage by_provider=ollama is the canary. Anything else for customer-shape traffic = regression. - ./scrum is a TOOL, not architecture. Outputs to data/_kb/ scrum_findings.jsonl. Findings inform dev, do NOT auto-fold into design docs. - Test code in main is actively being cleaned. Today: 12 files / ~2900 LOC removed (commits 6aafd41 + f4ebd22). Surface more candidates, don't auto-delete unless clearly orphaned. The intent: future me (or future Claude session) reads STATE_OF_PLAY on cold-start, sees these entries, and doesn't re-make the same mistakes that drifted scope today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
f4ebd2278b |
remove 7 more orphaned experimental scripts from scripts/
Continuing the test-code-in-main cleanup. These are sequential mode-runner experiment passes (2/3/4/5) that completed and whose findings were captured in pathway_memory + the matrix index — the scripts themselves are dead weight. Plus two one-off probe scripts. Removed (all 0 refs in production code or automation): - mode_pass2_corpus_sweep.ts — 2026-04 corpus sweep experiment - mode_pass3_variance.ts — variance measurement run - mode_pass4_staffing.ts — staffing-domain pass - mode_pass5_summarize.ts — summarization variance - mode_pass5_variance_paid.ts — paid-model variance - overnight_proof.sh — overnight stress probe (output in logs/) - ab_t3_test.sh — T3 overseer A/B test (output captured in KB) Verified: 0 references in package.json / justfile / Makefile / any active .ts/.rs/.sh file. Two mentions remain in docs/recon and docs/MODE_RUNNER_ TUNING_PLAN — those are historical design-doc references, not consumers. KEPT in scripts/ (have live consumers OR are runtime tools): - mode_experiment.ts (14 refs), mode_compare.ts (7 refs) - lance_smoke.sh, build_*_corpus.ts, staffing_demo.py, lance_tune.py, generate_demo.py, generate_workers.py, copilot.py, kb_measure.py, kb_staffer_report.py, analyze_chicago_contracts.ts, dump_raw_corpus.sh, check_phase44_callers.sh, autonomous_agent.py, build_answers_corpus.ts, build_lakehouse_corpus.ts, build_scrum_findings_corpus.ts, build_symbols_corpus.ts, e2e_pipeline_check.sh, scale_test.py, scale_10m_test.sh, run_staffer_demo.sh, stress_test.py Build clean. If any of these are needed back: git show HEAD~1 -- scripts/<file> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
6aafd41785 |
remove 5 orphaned dev experiments from tests/real-world/
Per J: "all of our test code ended up in the main." These are 5 one-time
dev experiments that were never wired into any automation and have zero
live consumers in the production code path. Deleting them.
Removed (1418 LOC total):
- enrich_prd_pipeline.ts (528 LOC) — Phase 21 architecture stress test
- nine_consecutive_audits.ts (185 LOC) — empirical study of audit compounding
- hard_task_escalation.ts (267 LOC) — escalation-ladder test (refs retired
cloud models qpt-oss:20b/120b)
- autonomous_loop.ts (214 LOC) — wrapper experiment around scrum_master
- consensus_reducer_design.ts (224 LOC) — N=3 design consultation; output JSON
referenced from pathway_memory.rs
comment but the script itself has
no consumer
Verified: 0 references in package.json / justfile / Makefile / any
production .ts/.rs/.sh file. The single mention from pathway_memory.rs
is a //! doc comment referencing the JSON output (data/_kb/
consensus_reducer_design_*.json), not the script. Build clean post-delete.
KEPT:
- scrum_master_pipeline.ts — referenced from observer.ts, vectord, scripts
- scrum_applier.ts — referenced from auditor schemas
If you need any of these back, they're in git history. cherry-pick or
git show HEAD~1 -- tests/real-world/<file>.ts will recover the source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
bb5a3b3f5e |
execution_loop: align overseer log/KB strings with reverted local route
Yesterday's revert (d054c0b) changed the API CALL from cloud to local but missed the LogEntry + KB row that record what model fired. Result: honest API call to qwen3.5:latest, dishonest log/KB rows saying "claude-opus-4-7". That's a real audit-trail integrity issue — the record didn't match reality. Fixed: - LogEntry "system" role label (line 663) - KB row's "model" field (line 685) Both now correctly show "qwen3.5:latest". Build + restart + smoke 10/10 green. Gateway healthy. Side note: the only remaining "claude-opus-4-7" mentions in this file are now in COMMENTS describing the v1 cloud route + the revert rationale — those are documentation, not log fields. Safe to keep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
a44ccde845 |
observer: overseer fallback label → qwen3.5:latest (matches reverted route)
Mirror of yesterday's execution_loop overseer revert (commit d054c0b). The observer logs an "overseer:<model>" endpoint string for analysis; when row.model is missing it falls back to a hardcoded label. PR #13 set that fallback to "claude-opus-4-7" — but the route now goes to local Ollama qwen3.5:latest, so the label was wrong. Trivial one-line fix, no behavior change. Just keeps observer's endpoint string honest when older rows from the cloud-routing window get re-analyzed. End-to-end verification of the local hot path (post-revert): BEFORE /v1/usage by_provider: [] AFTER /v1/usage by_provider: [{"k":"ollama","v":2}] → /v1/iterate fired 2 chat calls, both to local ollama → ZERO cloud requests (no kimi/openrouter/opencode/ollama_cloud) → API meter on cloud providers stays at 0 for customer requests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
2f5ca95875 |
scrum: real tool — auto-bundle current diff, run 3-reviewer scrum, push to KB
J's frustration captured: scrum was a TOOL meant to find gaps in the work
and push findings to a KB that informs how WE work on the code. It became
welded into architecture instead of staying a tool. This commits the tool
in the form J actually meant.
Usage:
./scrum auto-bundle origin/main..HEAD, auto-label
./scrum my_label same with explicit label
./scrum --staged bundle staged-only diff (pre-commit)
./scrum --since=COMMIT bundle from a specific commit
Output:
KB row → data/_kb/scrum_findings.jsonl (one row per scrum run)
Verdicts → reports/scrum/_evidence/<date>/verdicts/<label>_*.md
The KB row carries: timestamp, label, diff size, findings count,
convergent count, branch, head SHA, tally excerpt, paths to full
verdicts. Queryable via jq or DuckDB.
Cloud models (opus + kimi-k2 + qwen3-coder) are used here BY DESIGN
— 3-lineage cross-review needs distinct training corpora. This is dev
tooling, not the runtime hot path. PRD line 70 (no cloud APIs) applies
to customer requests, not to J's dev tools.
Tested live: ./scrum --since=HEAD~1 revert_only → 7 findings, 1 convergent
(INFO), all 3 reviewers VERDICT: ship for the revert commit.
KB row written:
{"label":"revert_only","findings_total":7,"findings_convergent":1,
"diff_bytes":5806,"branch":"demo/post-pr11-polish-2026-04-28",...}
What scrum does NOT do (intentionally):
- It does NOT auto-fold findings into architecture / design docs
- It does NOT block any commit/push
- It does NOT mutate any code
- It does NOT run as part of normal customer requests
It reports. J reads. J decides what to do with the findings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
d054c0b8b1 |
REVERT cloud routing on hot path — back to local Ollama per PRD line 70
PRD line 70: "Everything runs locally — no cloud APIs, total data privacy." Yesterday's PR #13 (feb638e) violated this by routing customer-facing inference paths to opencode + ollama_cloud + openrouter. Reverting the hot-path routes only; cloud providers stay configured in providers.toml for explicit dev-tool opt-in. Reverted: - modes.toml staffing_inference: kimi-k2.6 → qwen3.5:latest (local Ollama) - modes.toml doc_drift_check: gemini-3-flash-preview → qwen3.5:latest - execution_loop overseer: opencode/claude-opus-4-7 → ollama/qwen3.5:latest Was a paid Anthropic call on every overseer escalation; now local + free. Gateway compiles + restarts clean. Lance smoke 10/10. Live providers list unchanged (kimi/ollama_cloud/opencode/openrouter all still CONFIGURED; they just aren't ROUTED to from the staffing inference path anymore). This stops the API meter on customer requests. Cloud providers remain opt-in via explicit provider= caller hint, which the scrum tool + auditor pipeline + bot/propose use deliberately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
0c74b82fc8 |
phase 1.6 gate 4: REMOVE name → ethnicity / gender inference
Per docs/PHASE_1_6_BIPA_GATES.md Gate 4 + AUDIT_TRAIL_PRD §4 protected- attribute exclusion rule. The lookup tables + inference functions in search.html (3375-3499) and console.html (245-311) were dead code in the rendering path — headshot rendering disabled 2026-04-28 left these functions defined but unused. Removing them forecloses both Title VII discriminatory-feature-engineering AND BIPA biometric-information- derived-from-biometric-identifier arguments. Removed: - FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN - SURNAMES_HISPANIC, SURNAMES_SOUTH_ASIAN, SURNAMES_EAST_ASIAN, SURNAMES_MIDDLE_EASTERN, SURNAMES_BLACK - guessGenderFromFirstName(), guessEthnicityFromName(), guessEthnicityFromFirstName(), genderFor() From both search.html and console.html. Replacement: deprecation comment block referencing the BIPA gates doc. Verified: zero live consumers anywhere in mcp-server/. Searched for genderFor()/guessEthnicityFromName()/guessEthnicityFromFirstName()/ guessGenderFromFirstName() call sites — none remain. Per J 2026-05-03: this kind of test-code-leaked-into-main is exactly what J wants cleaned up. The face-pool inference was meant as a testing tool for synthetic icon generation but ended up as production-shape inference logic in the customer-facing UI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
cd440d4cee |
audit phase 1.6: BIPA pre-launch gates — block identity-service backfill
Per IDENTITY_SERVICE_DESIGN v3 §5 Step 0, Phase 1.6 is hard
prerequisite to identityd backfill. This doc specifies the 5 gates +
2 supporting deliverables that must ship before real-photo intake.
Five gates (BIPA §15 compliance):
1. Public retention schedule — counsel writes; engineering files+hash
2. Informed written consent — counsel writes template; engineering
wires identityd consent-status enforcement
3. Photo-upload endpoint with consent enforcement — POST /v1/identity/
subjects/{id}/photo with hard 403 when biometric_consent_status
!= 'given'; quarantined storage path; deepface output isolated
to identityd subjects table (not synthetic-face manifest)
4. Deprecate name → ethnicity inference (mcp-server/search.html
lookup tables removed; Phase 1.5 §1B finding closed)
5. Destruction runbook — operator-facing; ties to identityd
/erase endpoint with biometric-specific erasure path; daily
sweep job for biometric_retention_until expiry
Plus:
- Cryptographic attestation that no biometric data exists
pre-identityd (per v3-B11) — defends against
infrastructure-as-notice plaintiff argument
- Employee BIPA-handling training acknowledgment
Engineering effort: ~4-5 days (one week to stage everything ready).
Counsel effort: ~3-6 weeks calendar (review cycles dominate).
Calendar bottleneck is counsel, not engineering.
Phase 1.6 exit = 7 checked gates + signoffs. Until done, identityd
backfill cannot proceed (per identity service design v3 §5 Step 0).
5 open questions for J + counsel: photo-upload UX, consent
mechanism (DocuSign/click/paper), named operator list, named
counsel for sign-off, public privacy policy URL.
No code changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
8129ddd883 |
identity service: v3 amendments — second-pass scrum BUILD-WITH-CHANGES
Re-scrummed v2 across opus + kimi + gemini. All 3 verdict:
BUILD-WITH-CHANGES. v1 blockers verified RESOLVED. 12 new v2
findings folded as v3 amendments in §12.
Convergent v2 findings (≥2 reviewers):
v3-A1: mTLS CA root must NOT live in identityd (opus + gemini).
v3 fix: Vault PKI for CA, identityd as intermediate.
v3-A2: Dual-control public key registry must be tamper-evident
(opus + gemini). v3 fix: Vault KV with separate access
policies + server-issued nonces for replay protection.
Single-reviewer v3 amendments (10 more):
- B1: Step 8 fallback-to-SQL needs explicit 14-day time bound
- B2: NER drop-on-detect needs Prometheus alerting
- B3: legal-tier notification transport spec'd (signed Slack/email,
no PII in body, failure non-blocking)
- B4: Step 6 human review SLA flagged — ~7 months at 500/day for
~100k unknown rows; operational decision needed
- B5: Memory zeroing in Go is best-effort (Rust uses zeroize crate);
documented as not cryptographic-grade
- B6: purpose_definitions needs versioning + emergency revocation
(purpose_versions + purpose_revocations tables)
- B7: Cache invalidation needs erasure_generation atomicity
(subjects.erasure_generation int; gateway rejects stale-gen
cache hits) — replaces best-effort pub/sub
- B8: 15-min cooling-off period for dual-control issuance to
prevent emergency-bypass culture
- B9: NER calibrated test set with target recall ≥99.5% on
synthetic adversarial PII
- B10: S3 Object Lock in separate AWS account with write-only IAM;
root credentials held by external party
- B11: BIPA infrastructure-as-notice attestation in Phase 1.6 doc
- B12: Backup retention vs ciphertext-deletion erasure window
documented in RTBF runbook
Estimate revised v2 12-15d → v3 17-20d. Worth it — the cost is what
buys "I would build this" from 3 independent senior security
architects across 3 model lineages.
Must-have v3 items (block implementation): A1, A2, B1, B6, B7, B11.
Should-have (ship in Phase 5 if calendar tight): B2-B5, B8-B10, B12.
Re-scrum NOT recommended for v3 — diminishing returns; must-have
items are concrete fixes with clear acceptance criteria.
No code changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
298fadce41 |
identity service: v2 — fold cross-lineage scrum findings + 4 'would not build' blocker fixes
Scrummed v1 across opus + kimi + gemini lineages via the new model fleet. 3/3 reviewers said 'I would NOT build v1 as written.' 4 convergent blockers, all resolved in v2: 1. Migration order wrong — backfill before validation creates dark database; if backfill bug, no production traffic catches it. v2 inserts BIPA-prereq Step 0 + shadow-write before backfill + shadow-read before cutover. 9-step migration with cryptographic attestation of completeness at quarantine. 2. Master key on disk + legal token static file = 'security theater' per all 3. v2: HashiCorp Vault Transit / AWS KMS for KEK (not sealed file). Legal token: split-secret short-lived JWT (max 24h), dual-control issuance (J + counsel both sign), revocable in <60s. 3. consent_status='inferred_existing' is BIPA prima facie violation (kimi+gemini explicit). v2 backfill uses 'pending_backfill_review'; biometric data NEVER backfilled — separate consent stream. 4. Healthcare default 'general' = HIPAA exposure window for every misclassified subject. v2 default 'unknown' with fail-closed routing (treat unknown as healthcare-equivalent until classified by manual review). Auto-escalation to healthcare on resume_text pattern match. Plus 12 single-reviewer additions: - mTLS mandatory between gateway↔identityd (kimi) - External anchor for audit chain: S3 Object Lock 7-year compliance mode, hourly + on-event commits (all 3) - Audit-log signing key separate from encryption KEK (opus) - Field-level authorization via purpose_definitions table (kimi) - Per-row encryption keys deferred to Phase 7 (kimi simplification) - pii_access_log itself needs legal-tier read auth (opus) - Synchronous cache invalidation pub/sub on RTBF (opus) - Outbound NER pass for Langfuse defense-in-depth (opus TOCTOU) - model_version_hash per decision row (gemini) - /vertical minimal-disclosure endpoint (kimi HIPAA min-necessary) - Auto-escalation healthcare on resume_text pattern (kimi) - Rate limiting + token revocation list (opus) - Oracle tests in audit_parity.sh (kimi SOC2 CC4.1) Architecturally simplified per scrum: - Per-row encryption keys deferred to Phase 7 (single DEK + HSM- wrapped KEK + ciphertext deletion is equivalent practical erasure with less complexity) - PDF render deferred (JSON ships first) - Training-safe export deferred (not critical path) Estimated effort revised 8-10 → 12-15 days. Worth it — every addition was a 3/3-reviewer convergent finding. Re-scrum recommended before implementation starts to verify v2 addresses the v1 blockers. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
565ea4b32a |
audit phase 2: IDENTITY_SERVICE_DESIGN.md — full design doc
Incorporates J's confirmed answers (2026-05-03): - separate daemon (identityd) on :3225 / :4225 - signed JSON with PDF render for legal export - legal-only credential separate from admin token - Langfuse self-hosted (drops cross-border concern) - EU placeholder fields, not enforced - healthcare vertical routing — local-only models for healthcare PHI - training-safe export with hashed pseudonyms Plus Phase 1 + 1.5 findings + scrum-driven priorities: - UUID v7 candidate_id (drops kimi enumeration risk) - per-row encryption with per-subject keys (crypto-erasure target) - pii_access_log with Merkle-style integrity hash chain (FRE 901) - subject_id top-level promotion in all JSONL sinks - Langfuse boundary redaction layer (scrum C2 priority) - adverse-impact comparator pool in audit response (scrum C3) - BIPA-specific consent + retention metadata (scrum C4) - vertical detection at gateway boundary (J answer 10) Implementation single-language: Go (one identityd, both runtimes call it via HTTP). Postgres backing store, isolated schema. Master key in sealed file v1, vault migration path documented. 8-step migration path: stand up empty → backfill from parquet → behind feature flag → cut over reads incrementally → quarantine PII columns in workers_500k. Each step its own commit + gate + rollback. 6 open questions for J before implementation: master key location, Postgres shared vs isolated, vertical backfill default, legal token issuance procedure, crypto-erasure sweep cadence, EU enforcement timeline. Estimated 8-10 working days total. Largest single phase in the audit program. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
fd429f4185 |
audit phase 1.5: BIPA schema audit + outcomes.jsonl content sample
Two follow-up walks per AUDIT_PHASE_1_DISCOVERY §10/C4 + gemini scrum flag. Read-only. No code changes. BIPA findings: - scripts/staffing/tag_face_pool.py uses deepface to extract gender + race + age from face images. Output persists to data/headshots/ manifest.jsonl. For synthetic faces this is fine; for real candidate photos this becomes a regulated biometric database (740 ILCS 14/10). - mcp-server/index.ts:1408 ComfyUI prompt EXPLICITLY embeds protected attributes (age + race + gender) into model prompt — system-level encoding of protected-attribute features into AI workflow. - mcp-server/search.html:3375-3432 has hard-coded FEMALE_NAMES / MALE_NAMES / NAMES_HISPANIC / SURNAMES_* lookup tables — name-based ethnicity inference. Title VII / disparate-impact risk separate from BIPA. - data/headshots/manifest.jsonl is TRACKED IN GIT today (synthetic classifications). For real photos, this would be biometric data in version control — serious failure. - No consent flow, no public retention schedule, no deletion procedure, no employee training documented. All required by BIPA §15(a)/(b) before real-photo intake. outcomes.jsonl sample: - 39/101 rows persist candidate names in fills[*].name field today - Sample names: "Carmen I. Garcia", "Jamal Z. Jones", "Jacob N. Patel" (synthetic but real shape) - 0 hits for "culture fit" / "communication" / etc proxy phrases — synthetic data doesn't generate them. When real models reason about real candidates, they will. Append-only persistence makes RTBF cryptographic-erasure-only. Recommends Phase 1.6 (NEW) — BIPA pre-launch gates between Phase 1.5 and Phase 2: BIPA_COMPLIANCE_POLICY.md, consent gate at upload endpoint, quarantine real-photo classifications to data/biometric/, deprecate name->ethnicity lookup tables, unit test that synthetic manifest stays synthetic. 4-8 hours of design + one code commit. 5 open questions for J: where do real photos enter, will deepface tagging path stay for real photos, consent UX, retention duration floor, designated privacy officer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
64bda21614 |
audit PRD: J answered 5 open questions — fold into §10, revise phase plan
Conversation 2026-05-03 — J confirmed: - Photos/video YES → BIPA in full force ($1k-$5k per violation) - Langfuse self-hosted → drops GDPR Art. 44 cross-border concern - EU not in scope now but placeholder needed → design EU-compatible - Healthcare vertical YES → HIPAA BAA needed with model providers, PHI redaction at gateway boundary OR local-only routing for those requests, vertical-detection at boundary is Phase 2 requirement - Training/RAG MAY re-run on outcomes → design as if it will, training- safe export interface needed, crypto-erasure becomes load-bearing evidence chain §10 updated with answered/pending status per question. New §10.5 "Effect on phase plan" introduces: - Phase 1.5 (NEW) — BIPA photo/video schema audit + Langfuse boundary scoping + outcomes.jsonl content sample, BEFORE Phase 2 design - Phase 2 design must now include: EU-placeholder fields, vertical detection, training-safe export, BIPA consent metadata - Phase 9 rehearsal must cover discrimination + BIPA + healthcare PHI 3 questions still pending J's call before Phase 2 design ships: identity service daemon vs in-process, JSON vs signed PDF for legal export, audit endpoint auth model. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
627a5f0c3d |
audit phase 1: §10 scrum-review findings + walk back §1F over-claim
Ran cross-lineage scrum on the discovery doc with the new model fleet
(opus + kimi-k2.6 + gemini-3-flash via Go gateway :4110, custom
"senior security architect" prompt). 3/3 reviewers responded with
substantive 800-1200 word reviews. Saved at /tmp/audit_scrum/.
5 convergent findings (≥2 reviewers) added as §10/C1-C5:
C1. §1F matrix-indexer "good for audit defensibility" claim is over-
claimed — walked back in TL;DR. Trace bodies unverified; treat as
SUSPECTED PII sink until §8.1 sampling completes.
C2. §1E (Langfuse) is the most dangerous leak — fix FIRST, ahead of
view-routing. Boundary-crossing leak (GDPR Art. 44 / CPRA sale /
SOC2 disposal). All 3 reviewers converge on this priority.
C3. Discrimination defense requires the FULL CANDIDATE POOL, not just
fills. EEOC UGESP (1978): need adverse-impact stats on everyone
who could have been picked. Phase 1 worked example missed this.
C4. BIPA / biometric exposure understated in findings (in PRD §10.5
but not translated to actionables). $1k-$5k per-violation regime.
C5. candidate_id must be promoted to top-level field in all JSONL
sinks. Grepping natural-language strings is not defensible audit
strategy. 3/3 reviewers converge.
11 single-reviewer high-value catches added as §10 single-reviewer
section: opus on LLM provider egress (8th PII path), Art. 22 right-
to-explanation, special-category data, DPIA/ROPA/DPA inventory; kimi
on sequential ID enumeration risk, Langfuse retention config, CCPA
de-identified-in-place vs crypto-shred, Bun common-mode failure,
cryptographic audit-trail integrity (Merkle/FRE 901), HIPAA BAA,
revised SELECT * effort estimate; gemini on data residency, "culture
fit" reasoning proxies, comparator-pool snapshot.
§9 reordered: sample first → defense-layer second → Langfuse
boundary third (was view-routing first per original draft;
boundary-crossing leak is higher priority per scrum).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
505ea93726 |
audit phase 1: discovery walk complete — subject + PII surface map
Read-only walk of both runtimes per AUDIT_TRAIL_PRD.md §8 phase 1.
Fills "UNKNOWN" cells in PRD §3 + §7 with file:line evidence.
Headline findings:
- candidates_safe + workers_safe views EXIST as a defense layer but
are BYPASSED — tool registry SQL templates query raw tables
- PII traverses 7+ persistence/transmission paths per fill scenario:
SQL → tool_result → LogEntry → /v1/respond → Langfuse → outcomes.jsonl
→ overseer_corrections.jsonl
- candidate_id is stable but co-located with PII in workers_500k.parquet
(no separate identity service)
- /audit/subject/{id} endpoint does not exist
- Append-only persistence is universal — RTBF requires crypto-erasure
- Pathway memory is structurally subject-agnostic in fingerprints
(defensive); trace bodies may leak PII (needs sampling)
- Go side mirrors Rust PII shape — parity in the leak too
- Worked example (John Martinez audit today): NOT POSSIBLE to produce
complete-and-defensible response
Recommends 4 cheap high-value moves before Phase 2 design starts:
defense-layer enforcement (rewrite 3 SQL templates to _safe views),
sample state.json/Langfuse to confirm pathway memory is clean, walk
Bun mcp-server tool surface, schema-audit for protected-attribute
proxies. None are commitments — J's call.
No code changes in this commit. Companion to AUDIT_TRAIL_PRD.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
b2d717ae44 |
audit PRD: add §10.5 jurisdictional surface (IL + IN, federal, SOC2)
J flagged that the staffing system targets Chicago + Indiana — added a jurisdictional checklist section to the audit-trail PRD so counsel has a working starting point. Covered: - Federal: Title VII, ADEA, ADA, EEOC, OFCCP, FCRA, Section 1981 - Illinois: BIPA (high risk if any candidate photos), AI Video Interview Act (820 ILCS 42), Illinois Human Rights Act (broader than Title VII), PIPA breach notification, Day and Temporary Labor Services Act (directly applies — staffing industry-specific recordkeeping), Cook County + City of Chicago Human Rights Ordinances (additional protected classes including source of income, parental status, credit history) - Indiana: Data Breach Disclosure, Civil Rights Law (lighter than IL), Genetic Information Privacy Act - SOC 2 Type II as the typical SaaS sale gate (Privacy + Security TSCs most relevant; 6-9 month effort to first report) - HIPAA / PCI / ISO 27001 noted as out of current scope but flagged Phase reordering implications captured: - BIPA risk on real candidate photos may need to be resolved BEFORE audit-trail work (class-action exposure) - SOC 2 Type II prep runs in parallel, not after - IL Day and Temporary Labor Services recordkeeping may override our proposed 4-year retention SLA 7 open questions added that counsel must answer before the §8 phases can be locked in. Document is explicit (multiple times) that this is NOT legal advice — a research-grade checklist for J's counsel conversation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
c170ebc86e |
docs: AUDIT_TRAIL_PRD — production-readiness gate for staffing client
J flagged that smoke + parity tests prove the surface compiles, NOT that an audit response can be produced for a specific person — and the staffing client won't sign without defensible discrimination-claim response capability. New docs/AUDIT_TRAIL_PRD.md captures: - worked example: John Martinez at Warehouse B requests audit - subject audit response output format (per-decision row schema) - surface map: where decisions happen today, where the gaps are - PII handling rules (tokenization, protected-attribute exclusion, inferred-attribute risk) - identity service design intent (separate daemon, audited reads) - retention + right-to-be-forgotten policy intent - 9-phase implementation sequence with explicit per-phase exit criteria - cross-runtime requirement (both Rust + Go must satisfy) - 7 open questions blocking phase 2+ that need J's call STATE_OF_PLAY + PRD updated with explicit "production-ready blocker" section pointing at the new doc. The "substrate is shipped" framing gets a caveat: substrate ≠ production-ready until audit phase 9 exits. No code changes. This is the planning artifact J asked for before we start building. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
5368aca4d4 |
docs: sync ADR-019 + PRD + DECISIONS with 2026-05-02 substrate changes
ADR-019: closed the "re-bench when 10M corpus exists" follow-up. Added "Follow-up: 10M re-bench (2026-05-02)" section with the post-fix numbers (search ~20ms warm / ~46ms cold, doc-fetch ~5ms post-btree). Documented the lance-bench-bypassing-IndexMeta bug + 2-layer fix + gauntlet (7 unit + 12 sanitize + 10 smoke probes). Reframes the strategic question as "Lance vs Parquet+HNSW-with-spilling" since HNSW doesn't fit RAM at 10M. DECISIONS: added ADR-022 — drop Python sidecar from Rust hot path. Captures the rationale (236× embed perf gap was pure overhead), co-shipped LRU cache, dev-only Python that survives, cross-runtime parity verification, and the operator runbook signal (ps -ef ABSENT post-deploy). PRD: updated AI Boundary table line + aibridge crate description to reflect direct Ollama path (was: Python FastAPI sidecar → Ollama). Both lines reference ADR-022 for the full rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
e9d17f7d5a |
sanitize: drop over-broad path-missing branch + UTF-8-safe redaction
Re-scrum of yesterday's sanitizer fix surfaced 2 more real bugs in the fix itself (opus, both WARN, neither caught by kimi/qwen): W1 (service.rs:1949) — `mentions_path_missing` standalone branch was too aggressive. A registry-internal error like "/root/.cargo/.../x.rs: no such file or directory" would 404 because it triggers without dataset context. That's a real 500. Dropped the standalone branch; require dataset context AND missing-shape phrase. Lance's actual "Dataset at path X was not found" still satisfies it. W2 (service.rs:2018) — `out.push(bytes[i] as char)` corrupted multi-byte UTF-8 by casting raw bytes to char (only sound for ASCII < 128). A path containing user-supplied non-ASCII names produced Latin-1 mojibake. Rewrote redact_paths to track byte indices and emit unmatched runs as &str slices via push_str(&s[range]) — preserves multi-byte sequences verbatim. Step advance is now per-char, not per-byte, via small utf8_char_len helper. Two new regression tests: - is_not_found_does_not_match_unrelated_path_missing - redact_preserves_multibyte_utf8 (uses 工作 + café in input) 12/12 sanitize tests PASS. Smoke 10/10 PASS. Loop closure for opus re-scrum on the 2026-05-02 fix bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
ac7c996596 |
sweep up scrum WARNs — model const, stale config, temp_path entropy, smoke gate
Four findings deferred from the 2026-05-02 scrum, all 1-5 line fixes: W1 (kimi WARN @ scrum_master_pipeline.ts:1143) — `gemini-3-flash-preview` hardcoded twice in MAP and REDUCE phases. Extracted TREE_SPLIT_MODEL + TREE_SPLIT_PROVIDER constants near the existing config block. Diverging the two would break tree-split coherence (per-shard digests must come from the same model the reducer collapses). W2 (qwen WARN @ providers.toml:30) — stale `kimi-k2:1t` reference in operator-facing comments after PR #13 noted it's upstream-broken. Reframed as historical context ("was X here pre-2026-05-03 — that model is broken") so future operators don't paste-route from the comment. W3 (opus WARN @ vectord-lance/src/lib.rs:622) — temp_path() entropy was only pid+nanos, which collide under tokio scheduling when multiple tests in the same cargo process create temp dirs back-to-back. Added per-process AtomicU64 sequence counter — guarantees uniqueness regardless of clock. W4 (opus INFO @ scripts/lance_smoke.sh:38) — `|| echo '{}'` swallowed curl transport failures (gateway down, network broken, timeout), surfacing as misleading "no method field" jq errors at the next probe. Now captures $? separately, gates a "curl reachable" probe, and only falls back to empty body for the dependent jq parse. Smoke went 9 → 10 probes. Verified: vectord-lance 7/7 tests PASS, gateway cargo check clean, lance_smoke.sh 10/10 PASS against live gateway. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
root
|
7bb66f08c3 |
lance: scrum-driven sanitizer + smoke-gate fixes (opus 2026-05-02 BLOCK)
Some checks failed
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across"
Cross-lineage scrum on the lance wave (4 bundles, 33 distinct findings)
surfaced 1 real BLOCK and 2 real WARNs from opus that the kimi/qwen
lineages missed. Per feedback_cross_lineage_review.md, opus is the
load-bearing reviewer; cross-lineage convergence is noise unless verified.
BLOCK fix — sanitize_lance_err path-stripping was unsound:
err.split("/home/").next().unwrap_or(&err)
returns Some("") when err STARTS with "/home/", erasing the entire
message. Replaced truncation with redact_paths() — a hand-rolled scanner
that walks the input once, replacing path-shaped substrings with
[REDACTED] while preserving surrounding error context. Catches:
- absolute paths under /root/.cargo, /home, /var, /tmp, /etc, /usr, /opt
- relative variants (Lance occasionally strips leading slash —
observed live "Dataset at path home/profit/lakehouse/data/lance/x
was not found")
- multiple occurrences in one error
- preserves quote/comma/whitespace terminators
WARN fix #1 — is_not_found heuristic was too broad:
lower.contains("not found")
caught real 500s like "column not found", "field not found in schema".
Narrowed to require dataset-shape phrasing AND exclude the
column/field/schema patterns explicitly.
WARN fix #2 — lance_smoke.sh `grep -qvE` was an unsound regression gate.
bash -c "echo '$BODY' | grep -qvE 'pat'"
With -v -q, exits 0 if ANY line lacks the pattern — so a multi-line
body with one leak line + any clean line FALSE-PASSES. Replaced with
the correct "pattern absent" form: `! grep -qE 'pat'`. Also expanded
the pattern set (added /var/, /tmp/) since the scrum surfaced these
as additional leak vectors.
Also unblocks pre-existing pathway_memory test compile error (stale
PathwayTrace init missing 6 Mem0-versioning fields added in 6ac7f61).
Tests filled in with sensible defaults — needed to run sanitize_tests.
10/10 new sanitize tests pass. Smoke 9/9 PASS against rebuilt+restarted
gateway. Live missing-index probe now returns:
"lance dataset not found: no-such-11205" + HTTP 404
(was: leaked absolute paths + HTTP 500 → leaked absolute and relative
paths post-first-fix → clean message + 404 now.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
root
|
a294a61ee4 |
Merge remote-tracking branch 'origin/main' into demo/post-pr11-polish-2026-04-28
Some checks failed
lakehouse/auditor 9 blocking issues: cloud: claim not backed — "Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across"
|