docs: STATE_OF_PLAY — record search index-alignment fix + P1-already-done

- New 2026-06-30 wave: the ethereal_workers↔workers_500k_v1 misalignment fix (8 sites,
  commits 423400e/35a80a7/1570306), verified live after mcp-server restart
- Flag that demo-plan P1 (search-box filter) is already done — the P1 section is stale
- Note the untouched 2026-05-30 uncommitted WIP + newer untracked planning docs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claw 2026-06-30 22:50:31 -05:00
parent 15703063ea
commit 7dddb98f4b

View File

@ -1,12 +1,52 @@
# STATE OF PLAY — Lakehouse # STATE OF PLAY — Lakehouse
**Last verified:** 2026-05-05 morning CDT **Last verified:** 2026-06-30 (search index-alignment fix live on :3700) · biometric substrate 2026-05-05
**Verified by:** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory. **Verified by (2026-06-30):** live probe after mcp-server restart — `/match` + `/search` with `state='IL'` return only IL workers (new PID on :3700). Not memory.
**Verified by (2026-05-05):** live probe (`/audit/health` 200, `/biometric/subject/{id}/erase` 21-test substrate + `/audit/subject/{id}` legal-tier endpoint live verified against WORKER-100; new `verify_biometric_erasure.sh` + `biometric_destruction_report.sh` + `bundle_counsel_packet.sh` smoke-tested clean against live data) — not memory.
> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes. > **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
--- ---
## WHAT LANDED 2026-06-30 (search index-alignment sweep + P1-already-done finding)
**Fixed a systemic hybrid-search bug.** Every `/vectors/hybrid` call that paired the
`ethereal_workers` SQL dataset (10K rows) with the `workers_500k_v1` vector index (50K,
a *different* worker set) returned wrong-entity results: same `worker_id` = different
person in each index, so a `state='IL'` query surfaced a **KY** worker flagged
`sql_verified:true`. Root cause: the default index didn't share an id-space with the
default dataset.
Swept **8 call sites** in `mcp-server/index.ts` to `ethereal_workers_v1` (10K, same
id-space as the SQL table). Aligned sites (`filter_dataset: workers_500k`) and pure-RAG
sites (`/vectors/rag`) were correctly left alone.
| Commit | Sites |
|---|---|
| `423400e` | `/search` HTTP handler |
| `35a80a7` | `search_workers` tool · `match_contract` tool · `/match` HTTP handler |
| `1570306` | 4 intelligence routes (similar-workers, role-discovery, smart_search, scenario-find) |
**Verified live** after an mcp-server restart (new PID on :3700): `/match` and `/search`
with `state='IL'` return only IL workers (Champaign/Springfield/Urbana); `state='WI'`
returns WI. Grep confirms no `workers_500k_v1` index remains paired with `ethereal_workers`.
Server boots clean. Only `mcp-server/index.ts` was touched.
**Finding — demo-plan P1 was already done.** The "search box that actually filters" item
in this doc's CURRENT PLAN (P1, from the 2026-04-27 demo audit) is **stale**. The UI now
posts structured `state`/`role` to the evolved `/intelligence/chat`, which returns real
`sql_results` (IL Forklift → 167 matches, correct workers; WI → WI). The P1 section below
describes a bug that no longer exists — don't re-implement it.
**⚠ Uncommitted WIP left untouched (pre-existing, not from this session).**
`mcp-server/search.html` + `proof.html` (edited 2026-05-30; proof.html restructured into
chapters) and a `biometric_endpoint.rs` comment cleanup remain uncommitted in the working
tree. Also untracked and newer than this doc: `RECOVERY_PLAN_2026-05-05.md`,
`mcp-server/UX_FIX_PLAN.md`, `docs/scope-sentinel/`. Someone should reconcile these — they
imply planning context this file doesn't yet capture.
---
## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard) ## WHAT LANDED 2026-05-05 (afternoon wave — full BIPA lifecycle endpoints + UIs + ops dashboard)
After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`. After the morning's doc-reconciliation wave (described below), the afternoon wave shipped the full **operational lifecycle** for biometric data. Phase 1.6 now has the complete end-to-end runtime: candidate consent flow → photo upload → withdrawal → retention sweep flag → erase. Five commits, all on `demo/post-pr11-polish-2026-04-28`.