Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs
Worker cards now ship a real photo per person instead of monogram tiles:
- fetch_face_pool.py pulls 1000 faces from thispersondoesnotexist.com
- tag_face_pool.py runs deepface for gender/race/age, excludes <22yo
- manifest.jsonl: 952 servable, gender/race buckets populated
- /headshots/_thumbs/ pre-resized to 384px webp (587KB -> 11KB,
60x smaller; without this Chrome's parallel-connection budget
drops ~75% of tiles in a 40-card grid)
- /headshots/:key gender x race x age intersection bucketing with
gender-only fallback when intersection is sparse
- /headshots/generate/:key ComfyUI on-demand for the contractor
profile spotlight (cold ~1.5s, cached ~1ms; worker-derived
djb2 seed makes faces deterministic-per-worker but unique
across workers sharing the same prompt)
- serve_imagegen.py _cache_key() now includes seed (was caching
by prompt only -> 3 different worker seeds collapsed to 1
cached image; verified fix produces 3 distinct md5s)
- confidence-default name resolution: Xavier->man+hispanic,
Aisha->woman+black, etc. Every worker resolves to a bucket.
End-to-end: playwright run on /?q=forklift+operators+IL -> 21/21
cards loaded, 0 broken, all 384px webp.
Cache + binary pool gitignored; manifest tracked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
parent
10ed3bc630
commit
a3b65f314e
6
.gitignore
vendored
6
.gitignore
vendored
@ -6,5 +6,9 @@ __pycache__/
|
|||||||
*.pyc
|
*.pyc
|
||||||
|
|
||||||
# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
|
# Headshot pool — binary face JPGs are fetched by scripts/staffing/fetch_face_pool.py
|
||||||
# (synthetic StyleGAN, ~100MB for 200 faces). Manifest + fetch script are tracked.
|
# (synthetic StyleGAN, ~580MB for 1000 faces). Manifest + fetch script are tracked.
|
||||||
data/headshots/face_*.jpg
|
data/headshots/face_*.jpg
|
||||||
|
data/headshots/_thumbs/
|
||||||
|
# ComfyUI on-demand generated portraits (per-worker unique). Cached on first
|
||||||
|
# request; fully regeneratable via /headshots/generate/:key.
|
||||||
|
data/headshots_gen/
|
||||||
|
|||||||
239
STATE_OF_PLAY.md
Normal file
239
STATE_OF_PLAY.md
Normal file
@ -0,0 +1,239 @@
|
|||||||
|
# STATE OF PLAY — Lakehouse
|
||||||
|
|
||||||
|
**Last verified:** 2026-04-27 ~20:35 CDT
|
||||||
|
**Verified by:** live probe, not memory.
|
||||||
|
|
||||||
|
> **Read this FIRST.** When the user says "we're working on lakehouse," they mean the working code captured below — NOT what `git log` framed as "the cutover" or what memory snapshots from 2 days ago suggest. If memory contradicts this file, this file wins. Update it when something is verified working — not when a phase finishes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## VERIFIED WORKING RIGHT NOW
|
||||||
|
|
||||||
|
### The client demo (Staffing Co-Pilot)
|
||||||
|
|
||||||
|
**Public URL:** `https://devop.live/lakehouse/` — 200, "Staffing Co-Pilot" (159 KB SPA, leaflet maps, dark theme).
|
||||||
|
**Local URL:** `http://localhost:3700/` — same page, served by `mcp-server/index.ts` (PID 1271, started 09:48 CDT today).
|
||||||
|
|
||||||
|
**The staffers console** (the one the client was thoroughly impressed with):
|
||||||
|
- `https://devop.live/lakehouse/console` — 200, "Lakehouse — What Your Staffing System Would Do" (26 KB)
|
||||||
|
- Pulls project index via `/api/catalog/datasets` (36 datasets) + playbook memory via `/api/vectors/playbook_memory/stats` (4,701 entries with embeddings, real ops like *"fill: Maintenance Tech x2 in Milwaukee, WI"*)
|
||||||
|
|
||||||
|
Client-visible flow that works end-to-end on the public URL:
|
||||||
|
|
||||||
|
| Endpoint | Sample output |
|
||||||
|
|---|---|
|
||||||
|
| `GET /api/catalog/datasets` | 36 datasets indexed: timesheets 1M, call_log 800K, workers_500k 500K, email_log 500K, workers_100k 100K, candidates 100K, placements 50K, job_orders 15K, successful_playbooks_live 2,077 |
|
||||||
|
| `GET /api/vectors/playbook_memory/stats` | 4,701 fill operations with embeddings |
|
||||||
|
| `GET /system/summary` | 36 datasets, 2.98M rows, 60 indexes, 500K workers loaded, 1K candidates |
|
||||||
|
| `POST /intelligence/staffing_forecast` | 744 Production Workers needed in 30d, 11,281 bench (4,687 reliable), coverage 1,444%, risk=ok. Same for Electrician (need 32, bench 2,440) and Maintenance Tech (need 17, bench 5,004). |
|
||||||
|
| `POST /intelligence/permit_contracts` | permit `3442956` $500K → 3 Production Workers, 886-candidate pool, 95% fill, $36K gross. 5 more Chicago permits with 8 workers each, same pool, 95% fill, $96K each. |
|
||||||
|
| `POST /intelligence/market` | major Chicago permits ranked: $730M O'Hare, $615M 307 N Michigan, $580M casino, $445M Loop transit (real geo coords). |
|
||||||
|
| `POST /intelligence/permit_entities` | architects + contractors from permit contacts (e.g. "KACPRZYNSKI, ANDY", "SLS ELECTRICAL SERVICE"). |
|
||||||
|
| `POST /intelligence/activity` + `/intelligence/arch_signals` + `/intelligence/chat` | all 200 |
|
||||||
|
|
||||||
|
The demo tells the story: *"upcoming Chicago contracts → workers needed → coverage from the bench → architects/contractors involved → revenue and margin."* That's the "live data + anticipating contracts + complete workflow" pitch — working as of right now.
|
||||||
|
|
||||||
|
### Backend, verified live this session
|
||||||
|
|
||||||
|
| Surface | State |
|
||||||
|
|---|---|
|
||||||
|
| Gateway `:3100` | up, 4 providers configured, `/v1/health` 200 with 500K workers loaded |
|
||||||
|
| MCP server `:3700` (Co-Pilot demo) | up, all `/intelligence/*` endpoints respond |
|
||||||
|
| VCP UI `:3950` | started this session, `/data/*` 200, real numbers |
|
||||||
|
| Observer `:3800` | ring full (2,000/2,000) — older events evicted, query Langfuse for 24h-ago state |
|
||||||
|
| Sidecar `:3200` | up |
|
||||||
|
| Langfuse `:3001` | recording, `gw:/log` + `v1.chat:openrouter` traces visible |
|
||||||
|
| LLM Team UI `:5000` | up, only `extract` mode registered |
|
||||||
|
| OpenCode fleet | **40 models reachable through one `sk-*` key** (verified live `GET https://opencode.ai/zen/v1/models`) |
|
||||||
|
|
||||||
|
OpenCode catalog (live):
|
||||||
|
- Claude: opus-4-7, opus-4-6, opus-4-5, opus-4-1, sonnet-4-6, sonnet-4-5, sonnet-4, haiku-4-5
|
||||||
|
- GPT-5: 5.5-pro, 5.5, 5.4-pro, 5.4, 5.4-mini, 5.4-nano, 5.3-codex-spark, 5.3-codex, 5.2, 5.2-codex, 5.1-codex-max, 5.1-codex, 5.1-codex-mini, 5.1, 5-codex, 5-nano, 5
|
||||||
|
- Gemini: 3.1-pro, 3-flash
|
||||||
|
- GLM: 5.1, 5
|
||||||
|
- Minimax: m2.7, m2.5
|
||||||
|
- Kimi: k2.6, k2.5
|
||||||
|
- Qwen: 3.6-plus, 3.5-plus
|
||||||
|
- Other: BIG-PKL (was a typo-prone name in the catalog, model id starts with "big-pkl-something")
|
||||||
|
- Free tier: minimax-m2.5-free, hy3-preview-free, ling-2.6-flash-free, trinity-large-preview-free
|
||||||
|
|
||||||
|
### The substrate (frozen — do not re-architect)
|
||||||
|
|
||||||
|
- Distillation v1.0.0 at tag `e7636f2` — **145/145 bun tests pass, 22/22 acceptance, 16/16 audit-full**
|
||||||
|
- Output: `data/_kb/distilled_{facts,procedures,config_hints}.jsonl` + `data/vectors/distilled_{factual,procedural,config_hint}_v20260423102847.parquet`
|
||||||
|
- Auditor cross-lineage: Kimi K2.6 ↔ Haiku 4.5 alternation, Opus auto-promote on diffs >100k chars, **per-PR cap=3 with auto-reset on new head SHA**
|
||||||
|
- Pathway memory: 88 traces, 11/11 successful replays (probation gate crossed)
|
||||||
|
- Mode runner: 5 native modes; `codereview_isolation` is default; composed-corpus auto-downgrade verified Apr 26 (composed lost 5/5 vs isolation, p=0.031)
|
||||||
|
|
||||||
|
### Matrix indexer
|
||||||
|
|
||||||
|
30+ live corpora including:
|
||||||
|
- 5 versions of `workers_500k_v1..v9` (50K embedded chunks each)
|
||||||
|
- 11 batched 2K-row shards `w500k_b3..b17`
|
||||||
|
- `chicago_permits_v1` (3,420), `resumes_100k_v2` (100K candidates), `ethereal_workers_v1` (10K)
|
||||||
|
- `lakehouse_arch_v1` (2,119), `lakehouse_symbols_v1` (2,470), `lakehouse_answers_v1` (1,269), `scrum_findings_v1` (1,260)
|
||||||
|
- `kb_team_runs_v1` (12,693) + `kb_team_runs_agent` (4,407) — LLM-team play history embedded
|
||||||
|
- `distilled_factual_v20260423102507` (8) — distillation output
|
||||||
|
|
||||||
|
### Code health
|
||||||
|
|
||||||
|
- `cargo check --workspace` → **0 warnings, 0 errors**
|
||||||
|
- `bun test auditor + tests/distillation` → **145/145 pass**
|
||||||
|
- `ui/server.ts` + `auditor.ts` bundle clean
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## DO NOT RELITIGATE
|
||||||
|
|
||||||
|
- **PR #11 is merged into `origin/main` as `ed57eda`** — do not "still need to merge PR #11."
|
||||||
|
- **Distillation tag `distillation-v1.0.0` at `e7636f2` is FROZEN** — do not re-architect schemas, scorer rules, audit fixtures.
|
||||||
|
- **Kimi forensic HOLD verdict (2026-04-27) was 2/8 false + 6/8 latent** — do not re-debate, see `reports/kimi/audit-last-week-full.md`.
|
||||||
|
- **`candidates_safe` `vertical` column bug** — fixed at catalog metadata layer in commit `c3c9c21`. Do not "discover" it again.
|
||||||
|
- **Decisions A/B/C/D from `synthetic-data-gap-report.md`** — all four scripts shipped today (`d56f08e`, `940737d`, `c3c9c21`). Do not "ask J for approval."
|
||||||
|
- **`workers_500k.phone` type fixup** — already string. The fixup script is idempotent; running it is a no-op.
|
||||||
|
- **`client_workerskjkk` typo dataset** — was breaking every SQL query (catalog had it registered, file didn't exist). Removed via `DELETE /catalog/datasets/by-name/client_workerskjkk` this session. Do not re-add. Adding a startup gate that errors on unrecognized parquet names is the long-term fix per now.md Step 2C.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FIXES MADE THIS SESSION (2026-04-27 evening)
|
||||||
|
|
||||||
|
1. **`crates/gateway/src/v1/iterate.rs:93`** — `state` → `_state` (cleared the one cargo warning).
|
||||||
|
2. **`lakehouse-ui.service` (Dioxus)** — disabled. Was failing 7,242 times against a missing `target/dx/ui/debug/web/public` build dir. `systemctl stop && disable`.
|
||||||
|
3. **VCP UI on `:3950`** — started `bun run ui/server.ts` (PID 1162212, log `/tmp/lakehouse_ui.log`). `/data/*` endpoints now 200 with real data.
|
||||||
|
4. **`client_workerskjkk` catalog entry** — `DELETE /catalog/datasets/by-name/client_workerskjkk` removed the dead manifest. **This was the actual root cause** of `/system/summary` reporting `workers_500k_rows: 0` and the demo showing zero bench. Every SQL query was failing schema inference on the missing file before reaching its target table. Fixed → `workers_500k_rows: 500000`, `candidates_rows: 1000`, demo coverage flipped from "critical 0%" to actual percentages on devop.live/lakehouse.
|
||||||
|
|
||||||
|
## FIXES MADE THIS SESSION (2026-04-28 early — face pool)
|
||||||
|
|
||||||
|
5. **Synthetic StyleGAN face pool — 1000 faces, gender+race+age tagged.** `scripts/staffing/fetch_face_pool.py` fetches from thispersondoesnotexist.com; `scripts/staffing/tag_face_pool.py --min-age 22` runs deepface and excludes minors. `data/headshots/manifest.jsonl` now has gender (494 men / 458 women), race (caucasian 662 · east_asian 128 · hispanic 86 · middle_eastern 59 · black 14 · south_asian 3), age, and 48 minor exclusions. Server pool = 952 servable faces.
|
||||||
|
6. **`mcp-server/index.ts:1308` `/headshots/:key` route** — gender×race×age intersection bucketing with graceful fallback (gender-only → all). Same key always returns same face; different keys spread evenly.
|
||||||
|
7. **`/headshots/_thumbs/` pre-resized 384×384 webp** (60× smaller: 587KB → ~11KB). Without this, 40-card grids overran Chrome's parallel-connection budget and ~75% of tiles never finished decoding. Generated via parallel ffmpeg (`xargs -P 8`); `.gitignore`d.
|
||||||
|
8. **`mcp-server/search.html` + `console.html`** — dropped `img.loading='lazy'`. With 11KB thumbs, eager load is cheap (~500KB for 50 cards) and avoids the off-screen race that lazy decode produced.
|
||||||
|
9. **ComfyUI on-demand uniqueness — `serve_imagegen.py:32`** added `seed` to `_cache_key()` (was caching by prompt only — 3 different worker seeds collapsed to 1 cached image). Verified: seed=839185194/195/196 → 3 distinct md5s.
|
||||||
|
10. **`mcp-server/index.ts:1234` `/headshots/generate/:key`** — ComfyUI hot-path that derives a deterministic-per-worker seed via djb2-style hash; cold ~1.5s, cached ~1ms. Worker prompt format: `professional corporate headshot portrait of a {age}-year-old {race} {gender}, {role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`. Cache at `data/headshots_gen/` (gitignored, regeneratable).
|
||||||
|
11. **Confidence-default name resolution** in `search.html` — `genderFor()` and `guessEthnicityFromFirstName()` lookup tables (FEMALE_NAMES, MALE_NAMES, NAMES_HISPANIC, NAMES_BLACK, NAMES_SOUTH_ASIAN, NAMES_EAST_ASIAN, NAMES_MIDDLE_EASTERN). Xavier → man+hispanic, Aisha → woman+black, etc. Every worker resolves to a face-pool bucket.
|
||||||
|
|
||||||
|
End-to-end verified: playwright run on `https://devop.live/lakehouse/?q=forklift+operators+IL` → 21/21 cards loaded, 0 broken, all 384×384 webp thumbs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## OPEN — but not blocking the demo
|
||||||
|
|
||||||
|
| Item | What | When to act |
|
||||||
|
|---|---|---|
|
||||||
|
| `modes.toml` `staffing_inference.matrix_corpus` | still says `workers_500k_v8`. v9 in vector index is from Apr 17 (raw-sourced, not safe-view). The new `build_workers_v9.sh` rebuilds from `workers_safe`. | Run when you have 30+ min for the rebuild. |
|
||||||
|
| Open PRs #6, #7, #10 | sitting since Apr 22-24, auditor verdicts on disk at `data/_auditor/kimi_verdicts/{6,7,10}-*.json` | Read verdicts, decide reconcile/close. |
|
||||||
|
| `test/enrich-prd-pipeline` branch | 35 unmerged commits, includes more-evolved auditor/inference.ts (666 vs main's 580 lines), curation+fact-extractor wiring | Reconcile or formally archive — see `memory/project_unmerged_architecture_work.md`. |
|
||||||
|
| `federation-hnsw-trials` stash | Lance + S3/MinIO prototype, `aws-config` crate added, 708 insertions | Phase B from EXECUTION_PLAN.md — revisit when Parquet vector ceiling actually hurts. |
|
||||||
|
| `candidates` manifest drift | manifest 100K vs SQL 1K. Cosmetic. | Run a metadata resync if it matters. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## RUNTIME CHEATSHEET
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Verify the demo (public + local both work)
|
||||||
|
curl -sS https://devop.live/lakehouse/ # Co-Pilot HTML
|
||||||
|
curl -sS https://devop.live/lakehouse/console # staffers console
|
||||||
|
curl -sS -X POST https://devop.live/lakehouse/intelligence/staffing_forecast \
|
||||||
|
-d '{}' -H 'content-type: application/json' \
|
||||||
|
| jq '.forecast[] | {role, demand_workers, bench_total, coverage_pct, risk}'
|
||||||
|
|
||||||
|
# Restart sequence (after Rust changes)
|
||||||
|
sudo systemctl restart lakehouse.service # gateway :3100
|
||||||
|
sudo systemctl restart lakehouse-auditor # auditor daemon
|
||||||
|
sudo systemctl restart lakehouse-observer # observer :3800
|
||||||
|
# UI bun on :3950 is NOT systemd-managed (lakehouse-ui.service is disabled).
|
||||||
|
# Restart manually: kill <pid>; nohup bun run ui/server.ts > /tmp/lakehouse_ui.log 2>&1 &
|
||||||
|
|
||||||
|
# Health checks
|
||||||
|
curl -sS http://localhost:3100/v1/health | jq # workers_count, providers
|
||||||
|
curl -sS http://localhost:3100/vectors/pathway/stats | jq
|
||||||
|
curl -sS http://localhost:3100/v1/usage | jq # since-restart cost
|
||||||
|
curl -sS http://localhost:3700/system/summary | jq # dataset counts
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## VISION — what we're actually building (not what's done)
|
||||||
|
|
||||||
|
J's framing for the legacy staffing company:
|
||||||
|
|
||||||
|
- Pull live data, anticipate contracts based on Chicago permits → real architect/contractor associations, headcount, time period, money, scope.
|
||||||
|
- Hybrid + memory index → search large corpora cheaply.
|
||||||
|
- Email comes in → verify against contract; SMS comes in → alert when index changes.
|
||||||
|
- Real-time.
|
||||||
|
- Invent metrics nobody else has using the hybrid index.
|
||||||
|
- Next stage: workers download an app → geolocation clock-in → automatic responsiveness measurement, no user effort, with incentives for using it.
|
||||||
|
- Find people getting certificates (passive cert tracking).
|
||||||
|
- Pull union data → bring contracts that work for **employees**, not just employers.
|
||||||
|
- All metrics visible, nothing hidden, value-aligned with what each side actually needs.
|
||||||
|
|
||||||
|
If a future session is shaving away from this vision toward "fix the cutover" or "land Phase X," the vision wins. Phases are scaffolding for the vision, not the goal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CURRENT PLAN — fix the demo for the legacy staffing client
|
||||||
|
|
||||||
|
Built from playwright audit of the live demo (2026-04-27 evening). Each item ends in something the client can SEE, not internal cleanups.
|
||||||
|
|
||||||
|
**Demo state is anchored by git tag `demo-2026-04-27`** (commit `ed57eda`, the merge of PR #11). To restore code state: `git checkout demo-2026-04-27`. To restore runtime state: `DELETE /catalog/datasets/by-name/client_workerskjkk` (catalog hot-fix is not in git).
|
||||||
|
|
||||||
|
### P1 — Search box that actually filters (highest visible impact)
|
||||||
|
|
||||||
|
**Problem:** typing in `#sq` and pressing Enter fires `POST /intelligence/chat` with body `{"message":"<query>"}`. The state (`#sst`) and role (`#srl`) selects are ignored — never sent in the body. So every search returns a generic chat completion, never a SQL+vector hybrid filter against `workers_500k`. That is the "cached/generic response" the client sees.
|
||||||
|
|
||||||
|
**Fix:** in `mcp-server/search.html`, change the search-submit handler to call the real worker search endpoint with `{query, state, role, top_k}`. The MCP `search_workers` tool surface already exists; route the form there. Render returned worker rows in the existing card grid.
|
||||||
|
|
||||||
|
**Done when:** typing "forklift" + state IL + role "Forklift Operator" returns ≤ top_k IL Forklift Operators, and changing state to WI returns different workers.
|
||||||
|
|
||||||
|
### P2 — Contractor-name click → `/contractor` profile page
|
||||||
|
|
||||||
|
**Problem:** clicking a contractor name in any rendered card stays on `/lakehouse/`. URL doesn't change.
|
||||||
|
|
||||||
|
**Fix:** wrap contractor names in `<a href="/contractor?name=<encoded>">`. The page `mcp-server/contractor.html` (14.8 KB, "Contractor Profile · Staffing Co-Pilot") already exists at `/contractor` and the data endpoint `/intelligence/contractor_profile` already returns rich data.
|
||||||
|
|
||||||
|
**Then check contractor.html actually shows:** full history of every record the database has on that contractor + heat map of locations underneath + relevant info (per J 2026-04-27). If the page is incomplete, finish it. Otherwise just wire the link.
|
||||||
|
|
||||||
|
**Done when:** clicking "KACPRZYNSKI, ANDY" opens a profile with: every Chicago permit they're contact_1 or contact_2 on, a leaflet map with markers for each address, and any matched workers from prior placements at their sites.
|
||||||
|
|
||||||
|
### P3 — Substrate signal at the bottom shows the right numbers
|
||||||
|
|
||||||
|
**Problem:** J reports the bottom panel says "playbook memory empty, 80 traces 0 replies." Reality from the live endpoints: `/api/vectors/playbook_memory/stats` = 4,701 entries with embeddings; `/vectors/pathway/stats` = 88 traces, 11/11 replays.
|
||||||
|
|
||||||
|
**Fix:** find the renderer in search.html that builds the substrate signal panel; verify it's hitting the right endpoints and reading the right keys; fix shape mismatches.
|
||||||
|
|
||||||
|
**Done when:** bottom panel shows real numbers (4,701 playbooks, 88 traces, 11/11 replays) and references at least one specific recent operation from the playbook stats sample.
|
||||||
|
|
||||||
|
### P4 — Top nav reflects today's architecture
|
||||||
|
|
||||||
|
**Problem:** Walkthrough/Architecture/Spec/Onboard/Alerts/Workspaces tabs all return 200 but content is from old architecture. Doesn't mention: gateway scratchpad, memory indexer, ranker, mode runner, OpenCode 40-model fleet, distillation substrate, auditor cross-lineage.
|
||||||
|
|
||||||
|
**Fix:** rewrite `mcp-server/proof.html` (or add a single new page "What's running" that replaces Architecture+Spec) to describe what's actually shipped as of `demo-2026-04-27`. Keep one architecture page, drop redundancy. Either complete or hide Onboard/Alerts/Workspaces — J's call which.
|
||||||
|
|
||||||
|
**Done when:** the architecture page tells a non-technical reader, in 2 minutes, what each piece does in coordinator-relatable terms ("intern that read every email", not "3-stage adversarial inference pipeline").
|
||||||
|
|
||||||
|
### P5 — Caching for the project-index build_signal (J flagged unfinished)
|
||||||
|
|
||||||
|
**Problem:** "we never finished our caching for project index build signal it's not pulling new information." Need to find what `build_signal` refers to. Likely a scrum/auditor signal that should rebuild the `lakehouse_arch_v1` corpus on commit but isn't wired to.
|
||||||
|
|
||||||
|
**Fix:** identify the build-signal pipeline (likely in `auditor/` or `crates/vectord/`), wire its emit to a corpus rebuild, verify by making a test commit and watching the new chunk appear in `/vectors/indexes` for `lakehouse_arch_v1`.
|
||||||
|
|
||||||
|
**Done when:** committing a new file to `crates/` causes `lakehouse_arch_v1` chunk_count to increase within N minutes.
|
||||||
|
|
||||||
|
### P0 — Anchor the demo state (DONE)
|
||||||
|
|
||||||
|
Tagged `ed57eda` as `demo-2026-04-27`. Future sessions: `git checkout demo-2026-04-27` to land in this exact code state.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## EXECUTION ORDER
|
||||||
|
|
||||||
|
1. **P1 first** — biggest visible bug, ~30-60 min
|
||||||
|
2. **P2 next** — contractor click is the second-biggest "doesn't work" the client sees, ~20 min if profile is mostly done
|
||||||
|
3. **P3** — small fix, big "looks alive" win
|
||||||
|
4. **P4** — biggest scope; might split across sessions
|
||||||
|
5. **P5** — feature work, only after the visible bugs are fixed
|
||||||
|
|
||||||
|
Each item commits independently with the format `demo: P<n> — <one-line>` so the commit log doubles as a progress journal. After each merge to main, re-tag `demo-latest` to point at the new HEAD.
|
||||||
|
|
||||||
|
Stop here and let J pick which item to start with. Do not silently extend scope.
|
||||||
File diff suppressed because it is too large
Load Diff
@ -291,7 +291,8 @@ function workerRow(name, role, detail, opts){
|
|||||||
if(faceKey){
|
if(faceKey){
|
||||||
var img=document.createElement('img');
|
var img=document.createElement('img');
|
||||||
img.alt='';
|
img.alt='';
|
||||||
img.loading='lazy';
|
// Eager: 11KB thumbs make lazy unnecessary and lazy was racing
|
||||||
|
// playwright + retina-decode in field testing.
|
||||||
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + '?g='+gHint+'&e='+eHint;
|
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + '?g='+gHint+'&e='+eHint;
|
||||||
img.onerror=function(){ this.remove(); };
|
img.onerror=function(){ this.remove(); };
|
||||||
av.appendChild(img);
|
av.appendChild(img);
|
||||||
|
|||||||
@ -1225,6 +1225,77 @@ async function main() {
|
|||||||
// OSHA national, Chicago history, ticker chart, parent link,
|
// OSHA national, Chicago history, ticker chart, parent link,
|
||||||
// federal contracts, debarment, unions, training. Click any
|
// federal contracts, debarment, unions, training. Click any
|
||||||
// contractor name in a permit Entity Brief to land here.
|
// contractor name in a permit Entity Brief to land here.
|
||||||
|
// ComfyUI-generated portrait — every call is unique by (key,
|
||||||
|
// gender, race, age, role) tuple. First hit takes ~1.5s on
|
||||||
|
// the A4000; subsequent hits read from disk. Use this for
|
||||||
|
// contractor / profile modal where one worker gets the
|
||||||
|
// spotlight. NB: declared BEFORE the pool route so the prefix
|
||||||
|
// match doesn't intercept it.
|
||||||
|
if (url.pathname.startsWith("/headshots/generate/") && req.method === "GET") {
|
||||||
|
const key = decodeURIComponent(url.pathname.slice("/headshots/generate/".length));
|
||||||
|
if (!key) return new Response("missing key", { status: 400 });
|
||||||
|
const g = (url.searchParams.get("g") || "person").toLowerCase();
|
||||||
|
const r = (url.searchParams.get("e") || "").toLowerCase();
|
||||||
|
const role = (url.searchParams.get("role") || "warehouse worker").toLowerCase();
|
||||||
|
const age = parseInt(url.searchParams.get("age") || "32", 10) || 32;
|
||||||
|
const cacheKey = await crypto.subtle.digest(
|
||||||
|
"SHA-256",
|
||||||
|
new TextEncoder().encode(`${key}|${g}|${r}|${role}|${age}`)
|
||||||
|
).then((b) => Array.from(new Uint8Array(b)).map((x) => x.toString(16).padStart(2, "0")).join("").slice(0, 24));
|
||||||
|
const GEN_DIR = "/home/profit/lakehouse/data/headshots_gen";
|
||||||
|
await Bun.$`mkdir -p ${GEN_DIR}`.quiet();
|
||||||
|
const cachePath = `${GEN_DIR}/${cacheKey}.webp`;
|
||||||
|
const cached = Bun.file(cachePath);
|
||||||
|
if (await cached.exists()) {
|
||||||
|
return new Response(cached, {
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "image/webp",
|
||||||
|
"Cache-Control": "public, max-age=86400, immutable",
|
||||||
|
"X-Headshot-Source": "comfyui-cached",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const raceText = r === "hispanic" ? "Hispanic"
|
||||||
|
: r === "black" ? "Black"
|
||||||
|
: r === "south_asian" ? "South Asian"
|
||||||
|
: r === "east_asian" ? "East Asian"
|
||||||
|
: r === "middle_eastern" ? "Middle Eastern"
|
||||||
|
: "";
|
||||||
|
const genderText = g === "woman" ? "woman" : g === "man" ? "man" : "person";
|
||||||
|
const prompt = `professional corporate headshot portrait of a ${age}-year-old ${raceText} ${genderText}, ${role}, neutral expression, plain studio background, soft natural lighting, sharp focus, photorealistic, dslr`;
|
||||||
|
// Worker-derived seed — same input always picks the same
|
||||||
|
// pixel layout in StyleGAN2 latent space, so the face is
|
||||||
|
// deterministic per worker BUT distinct from any other
|
||||||
|
// worker that happens to share the same prompt. Without
|
||||||
|
// this, every (g, r, age, role) combo collapses to one face.
|
||||||
|
let seedHash = 0;
|
||||||
|
for (let i = 0; i < key.length; i++) seedHash = ((seedHash << 5) - seedHash + key.charCodeAt(i)) | 0;
|
||||||
|
const seed = Math.abs(seedHash) % 2147483647;
|
||||||
|
try {
|
||||||
|
const genResp = await fetch("http://localhost:3600/generate", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ prompt, width: 512, height: 512, steps: 8, seed }),
|
||||||
|
signal: AbortSignal.timeout(30000),
|
||||||
|
});
|
||||||
|
if (!genResp.ok) return new Response(`gen failed: ${genResp.status}`, { status: 502 });
|
||||||
|
const data: any = await genResp.json();
|
||||||
|
if (!data.image) return new Response("no image returned", { status: 502 });
|
||||||
|
const bytes = Uint8Array.from(atob(data.image), (c) => c.charCodeAt(0));
|
||||||
|
await Bun.write(cachePath, bytes);
|
||||||
|
return new Response(bytes, {
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "image/webp",
|
||||||
|
"Cache-Control": "public, max-age=86400, immutable",
|
||||||
|
"X-Headshot-Source": "comfyui-fresh",
|
||||||
|
"X-Headshot-Gen-Ms": String(data.time_ms || 0),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
} catch (e: any) {
|
||||||
|
return new Response(`gen error: ${e.message}`, { status: 502 });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Headshot pool — synthetic StyleGAN faces from
|
// Headshot pool — synthetic StyleGAN faces from
|
||||||
// thispersondoesnotexist.com fetched offline by
|
// thispersondoesnotexist.com fetched offline by
|
||||||
// scripts/staffing/fetch_face_pool.py. Deterministic mapping:
|
// scripts/staffing/fetch_face_pool.py. Deterministic mapping:
|
||||||
@ -1249,19 +1320,47 @@ async function main() {
|
|||||||
const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text();
|
const raw = await Bun.file(`${HEADSHOT_DIR}/manifest.jsonl`).text();
|
||||||
const lines = raw.trim().split("\n").filter(Boolean);
|
const lines = raw.trim().split("\n").filter(Boolean);
|
||||||
const all = lines.map((l) => JSON.parse(l));
|
const all = lines.map((l) => JSON.parse(l));
|
||||||
|
// Build (gender × race) buckets so a request that names
|
||||||
|
// both narrows to the intersection. Missing intersections
|
||||||
|
// fall back to gender-only, then race-only, then all.
|
||||||
|
const byGR: Record<string, any[]> = {};
|
||||||
|
const byG: Record<string, any[]> = { man: [], woman: [] };
|
||||||
|
const byR: Record<string, any[]> = {};
|
||||||
|
// Filter excluded faces (e.g. minors) from every bucket
|
||||||
|
// and from the all-pool. They never get served.
|
||||||
|
const adults = all.filter((r: any) => !r.excluded);
|
||||||
|
for (const r of adults) {
|
||||||
|
if (r.gender === "man" || r.gender === "woman") byG[r.gender].push(r);
|
||||||
|
if (r.race) {
|
||||||
|
byR[r.race] = byR[r.race] || [];
|
||||||
|
byR[r.race].push(r);
|
||||||
|
if (r.gender === "man" || r.gender === "woman") {
|
||||||
|
const k = r.gender + "/" + r.race;
|
||||||
|
byGR[k] = byGR[k] || [];
|
||||||
|
byGR[k].push(r);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
(globalThis as any)._faces = {
|
(globalThis as any)._faces = {
|
||||||
all,
|
all: adults,
|
||||||
man: all.filter((r: any) => r.gender === "man"),
|
byG, byR, byGR,
|
||||||
woman: all.filter((r: any) => r.gender === "woman"),
|
untagged: adults.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
|
||||||
untagged: all.filter((r: any) => !r.gender || (r.gender !== "man" && r.gender !== "woman")),
|
excluded_count: all.length - adults.length,
|
||||||
loaded_at: Date.now(),
|
loaded_at: Date.now(),
|
||||||
};
|
};
|
||||||
if (key === "__reload") {
|
if (key === "__reload") {
|
||||||
|
const byRSummary: Record<string, number> = {};
|
||||||
|
for (const k of Object.keys(byR)) byRSummary[k] = byR[k].length;
|
||||||
|
const byGRSummary: Record<string, number> = {};
|
||||||
|
for (const k of Object.keys(byGR)) byGRSummary[k] = byGR[k].length;
|
||||||
return Response.json({
|
return Response.json({
|
||||||
reloaded: true,
|
reloaded: true,
|
||||||
total: all.length,
|
total: all.length,
|
||||||
man: (globalThis as any)._faces.man.length,
|
excluded: all.length - adults.length,
|
||||||
woman: (globalThis as any)._faces.woman.length,
|
served_pool: adults.length,
|
||||||
|
by_gender: { man: byG.man.length, woman: byG.woman.length },
|
||||||
|
by_race: byRSummary,
|
||||||
|
by_gender_race: byGRSummary,
|
||||||
untagged: (globalThis as any)._faces.untagged.length,
|
untagged: (globalThis as any)._faces.untagged.length,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@ -1269,20 +1368,50 @@ async function main() {
|
|||||||
return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 });
|
return new Response(`face pool not available: ${e.message}. Run scripts/staffing/fetch_face_pool.py first.`, { status: 503 });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
const F = (globalThis as any)._faces as { all: any[]; man: any[]; woman: any[]; untagged: any[] };
|
const F = (globalThis as any)._faces as {
|
||||||
|
all: any[];
|
||||||
|
byG: Record<string, any[]>;
|
||||||
|
byR: Record<string, any[]>;
|
||||||
|
byGR: Record<string, any[]>;
|
||||||
|
untagged: any[];
|
||||||
|
};
|
||||||
if (!F || !F.all.length) {
|
if (!F || !F.all.length) {
|
||||||
return new Response("face pool empty", { status: 503 });
|
return new Response("face pool empty", { status: 503 });
|
||||||
}
|
}
|
||||||
// Pool selection: gender hint > full pool. If no gender match,
|
// Pool selection: try gender×race intersection first, then
|
||||||
// fall back to the full pool so the worker still gets a face.
|
// gender-only, then race-only, then full pool. Always returns
|
||||||
|
// a face so the worker card never falls back to the monogram.
|
||||||
|
const wantRace = url.searchParams.get("e") || "";
|
||||||
let pool = F.all;
|
let pool = F.all;
|
||||||
if (wantGender === "man" && F.man.length) pool = F.man;
|
if (wantGender && wantRace && F.byGR[wantGender + "/" + wantRace]?.length) {
|
||||||
else if (wantGender === "woman" && F.woman.length) pool = F.woman;
|
pool = F.byGR[wantGender + "/" + wantRace];
|
||||||
|
} else if (wantGender && F.byG[wantGender]?.length) {
|
||||||
|
pool = F.byG[wantGender];
|
||||||
|
} else if (wantRace && F.byR[wantRace]?.length) {
|
||||||
|
pool = F.byR[wantRace];
|
||||||
|
}
|
||||||
// Hash key → pool index. djb2-ish, fits any string.
|
// Hash key → pool index. djb2-ish, fits any string.
|
||||||
let h = 5381;
|
let h = 5381;
|
||||||
for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0;
|
for (let i = 0; i < key.length; i++) h = ((h << 5) + h + key.charCodeAt(i)) | 0;
|
||||||
const idx = Math.abs(h) % pool.length;
|
const idx = Math.abs(h) % pool.length;
|
||||||
const pick = pool[idx];
|
const pick = pool[idx];
|
||||||
|
// Prefer pre-resized webp thumb (~10KB) over native JPEG
|
||||||
|
// (~580KB). 60× smaller — without this, a 40-card grid
|
||||||
|
// overruns Chrome's parallel-connection budget and ~75% of
|
||||||
|
// tiles never finish decoding.
|
||||||
|
const thumbName = pick.file.replace(/\.jpg$/, ".webp");
|
||||||
|
const thumb = Bun.file(`${HEADSHOT_DIR}/_thumbs/${thumbName}`);
|
||||||
|
if (await thumb.exists()) {
|
||||||
|
return new Response(thumb, {
|
||||||
|
headers: {
|
||||||
|
"Content-Type": "image/webp",
|
||||||
|
"Cache-Control": "public, max-age=86400, immutable",
|
||||||
|
"X-Face-Pool-Idx": String(pick.id),
|
||||||
|
"X-Face-Pool-Gender": pick.gender || "untagged",
|
||||||
|
"X-Face-Pool-Variant": "thumb-384",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`);
|
const file = Bun.file(`${HEADSHOT_DIR}/${pick.file}`);
|
||||||
if (!(await file.exists())) {
|
if (!(await file.exists())) {
|
||||||
return new Response("face missing on disk", { status: 404 });
|
return new Response("face missing on disk", { status: 404 });
|
||||||
@ -1293,6 +1422,7 @@ async function main() {
|
|||||||
"Cache-Control": "public, max-age=86400, immutable",
|
"Cache-Control": "public, max-age=86400, immutable",
|
||||||
"X-Face-Pool-Idx": String(pick.id),
|
"X-Face-Pool-Idx": String(pick.id),
|
||||||
"X-Face-Pool-Gender": pick.gender || "untagged",
|
"X-Face-Pool-Gender": pick.gender || "untagged",
|
||||||
|
"X-Face-Pool-Variant": "native-1024",
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
@ -2397,7 +2397,9 @@ function addWorkerInsight(parent,name,detail,why,idx,highlight){
|
|||||||
if(faceKey){
|
if(faceKey){
|
||||||
var img=document.createElement('img');
|
var img=document.createElement('img');
|
||||||
img.alt='';
|
img.alt='';
|
||||||
img.loading='lazy';
|
// No lazy-loading: thumbs are 384x384 webp (~11KB) so eager
|
||||||
|
// load is cheap (~500KB for 50 cards) and avoids the off-screen
|
||||||
|
// tile flash + scroll-jitter that lazy decode produces here.
|
||||||
var qs = '?g=' + gHint + '&e=' + eHint;
|
var qs = '?g=' + gHint + '&e=' + eHint;
|
||||||
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + qs;
|
img.src = P + '/headshots/' + encodeURIComponent(faceKey) + qs;
|
||||||
img.onerror=function(){ this.remove(); };
|
img.onerror=function(){ this.remove(); };
|
||||||
|
|||||||
@ -29,8 +29,14 @@ CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
|||||||
WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json"
|
WORKFLOW_PATH = "/opt/ComfyUI/workflows/editorial_hero.json"
|
||||||
|
|
||||||
|
|
||||||
def _cache_key(prompt, width, height, steps):
|
def _cache_key(prompt, width, height, steps, seed=None):
|
||||||
return hashlib.sha256(f"{prompt}|{width}|{height}|{steps}".encode()).hexdigest()[:24]
|
# Include seed so callers can vary outputs deterministically without
|
||||||
|
# the proxy collapsing to a single cached image. None == legacy
|
||||||
|
# (omitted from the key for backward compatibility).
|
||||||
|
bits = f"{prompt}|{width}|{height}|{steps}"
|
||||||
|
if seed is not None:
|
||||||
|
bits += f"|{seed}"
|
||||||
|
return hashlib.sha256(bits.encode()).hexdigest()[:24]
|
||||||
|
|
||||||
def _cache_get(key):
|
def _cache_get(key):
|
||||||
fp = CACHE_DIR / f"{key}.webp"
|
fp = CACHE_DIR / f"{key}.webp"
|
||||||
@ -178,8 +184,9 @@ class ImageHandler(BaseHTTPRequestHandler):
|
|||||||
steps = min(max(int(body.get("steps", 50)), 1), 80)
|
steps = min(max(int(body.get("steps", 50)), 1), 80)
|
||||||
seed = body.get("seed")
|
seed = body.get("seed")
|
||||||
|
|
||||||
# Cache check
|
# Cache check — seed is part of the key so per-worker requests
|
||||||
key = _cache_key(prompt, width, height, steps)
|
# don't collapse to a single cached portrait.
|
||||||
|
key = _cache_key(prompt, width, height, steps, seed)
|
||||||
cached = _cache_get(key)
|
cached = _cache_get(key)
|
||||||
if cached:
|
if cached:
|
||||||
self._json(200, {"image": cached, "format": "webp", "width": width, "height": height,
|
self._json(200, {"image": cached, "format": "webp", "width": width, "height": height,
|
||||||
@ -210,6 +217,11 @@ class ImageHandler(BaseHTTPRequestHandler):
|
|||||||
|
|
||||||
elapsed_ms = int((time.time() - t0) * 1000)
|
elapsed_ms = int((time.time() - t0) * 1000)
|
||||||
img_b64 = base64.b64encode(img_bytes).decode()
|
img_b64 = base64.b64encode(img_bytes).decode()
|
||||||
|
# Recompute key with the actual seed used (when caller passed
|
||||||
|
# None, _comfyui_generate picks a random one and we want the
|
||||||
|
# cache to reflect that so re-requests with the same returned
|
||||||
|
# seed hit the disk).
|
||||||
|
key = _cache_key(prompt, width, height, steps, seed)
|
||||||
_cache_put(key, img_bytes)
|
_cache_put(key, img_bytes)
|
||||||
|
|
||||||
self._json(200, {
|
self._json(200, {
|
||||||
|
|||||||
169
scripts/staffing/tag_face_pool.py
Normal file
169
scripts/staffing/tag_face_pool.py
Normal file
@ -0,0 +1,169 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
tag_face_pool.py — run deepface gender + race classification over the
|
||||||
|
synthetic face pool produced by fetch_face_pool.py and rewrite
|
||||||
|
manifest.jsonl with `gender` (man / woman) and `race` (asian / black /
|
||||||
|
hispanic / indian / middle_eastern / white) tags.
|
||||||
|
|
||||||
|
Run with the venv that has deepface installed:
|
||||||
|
|
||||||
|
/home/profit/.local/share/deepface-venv/bin/python \
|
||||||
|
scripts/staffing/tag_face_pool.py
|
||||||
|
|
||||||
|
Idempotent: rows that already have BOTH gender and race tagged are
|
||||||
|
skipped. Pass --force to re-tag everything.
|
||||||
|
|
||||||
|
Mapping deepface buckets → /headshots/ ?e= values:
|
||||||
|
asian → split by manual region (deepface doesn't differentiate
|
||||||
|
East / South Asian; we lump as 'east_asian' since the
|
||||||
|
StyleGAN training set leans East Asian)
|
||||||
|
indian → south_asian
|
||||||
|
middle eastern → middle_eastern
|
||||||
|
black → black
|
||||||
|
hispanic → hispanic
|
||||||
|
white → caucasian
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
DEEPFACE_RACE_TO_HINT = {
|
||||||
|
"asian": "east_asian",
|
||||||
|
"indian": "south_asian",
|
||||||
|
"middle eastern": "middle_eastern",
|
||||||
|
"black": "black",
|
||||||
|
"latino hispanic": "hispanic",
|
||||||
|
"hispanic": "hispanic",
|
||||||
|
"white": "caucasian",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
p = argparse.ArgumentParser()
|
||||||
|
p.add_argument(
|
||||||
|
"--out",
|
||||||
|
default=os.path.join(os.path.dirname(__file__), "..", "..", "data", "headshots"),
|
||||||
|
)
|
||||||
|
p.add_argument("--force", action="store_true", help="re-tag rows that already have gender+race")
|
||||||
|
p.add_argument("--limit", type=int, default=0, help="cap how many faces to process this run (0 = all)")
|
||||||
|
p.add_argument("--min-age", type=int, default=22, help="exclude faces estimated below this age (kids/teens). Staffing context = legal-age workers only.")
|
||||||
|
args = p.parse_args()
|
||||||
|
|
||||||
|
out = os.path.realpath(args.out)
|
||||||
|
manifest_path = os.path.join(out, "manifest.jsonl")
|
||||||
|
if not os.path.exists(manifest_path):
|
||||||
|
print(f"manifest not found: {manifest_path}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"loading deepface (cold start ~10-15s for first model build)…")
|
||||||
|
from deepface import DeepFace # type: ignore
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
with open(manifest_path) as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
rows.append(json.loads(line))
|
||||||
|
print(f"manifest: {len(rows)} rows")
|
||||||
|
|
||||||
|
todo = [
|
||||||
|
r for r in rows
|
||||||
|
if args.force or r.get("gender") is None or r.get("race") is None or r.get("age") is None
|
||||||
|
]
|
||||||
|
if args.limit > 0:
|
||||||
|
todo = todo[: args.limit]
|
||||||
|
print(f"to tag: {len(todo)} faces")
|
||||||
|
|
||||||
|
if not todo:
|
||||||
|
print("nothing to do.")
|
||||||
|
return
|
||||||
|
|
||||||
|
counts_g = {}
|
||||||
|
counts_r = {}
|
||||||
|
failed = 0
|
||||||
|
t0 = time.time()
|
||||||
|
for i, r in enumerate(todo):
|
||||||
|
full = os.path.join(out, r["file"])
|
||||||
|
try:
|
||||||
|
ana = DeepFace.analyze(
|
||||||
|
img_path=full,
|
||||||
|
actions=["gender", "race", "age"],
|
||||||
|
enforce_detection=False,
|
||||||
|
silent=True,
|
||||||
|
)
|
||||||
|
if isinstance(ana, list):
|
||||||
|
ana = ana[0] if ana else {}
|
||||||
|
g_raw = (ana.get("dominant_gender") or "").lower().strip()
|
||||||
|
r["gender"] = (
|
||||||
|
"man" if g_raw.startswith("man") else
|
||||||
|
"woman" if g_raw.startswith("woman") else
|
||||||
|
None
|
||||||
|
)
|
||||||
|
r_raw = (ana.get("dominant_race") or "").lower().strip()
|
||||||
|
r["race"] = DEEPFACE_RACE_TO_HINT.get(r_raw, None)
|
||||||
|
if r["race"] is None and r_raw:
|
||||||
|
r["race_raw"] = r_raw
|
||||||
|
# Age estimation — exclude minors / teens. Staffing context
|
||||||
|
# uses adult workers only. Threshold is 22 by default
|
||||||
|
# (legal + a buffer because age estimation is noisy).
|
||||||
|
try:
|
||||||
|
age = int(round(float(ana.get("age") or 0)))
|
||||||
|
except Exception:
|
||||||
|
age = 0
|
||||||
|
r["age"] = age
|
||||||
|
if age and age < args.min_age:
|
||||||
|
r["excluded"] = "minor"
|
||||||
|
else:
|
||||||
|
r.pop("excluded", None)
|
||||||
|
counts_g[r["gender"] or "unknown"] = counts_g.get(r["gender"] or "unknown", 0) + 1
|
||||||
|
counts_r[r["race"] or r_raw or "unknown"] = counts_r.get(r["race"] or r_raw or "unknown", 0) + 1
|
||||||
|
except Exception as e:
|
||||||
|
r["tag_error"] = f"{type(e).__name__}: {e}"
|
||||||
|
failed += 1
|
||||||
|
if (i + 1) % 25 == 0 or (i + 1) == len(todo):
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
rate = (i + 1) / elapsed if elapsed > 0 else 0
|
||||||
|
eta = (len(todo) - i - 1) / rate if rate > 0 else 0
|
||||||
|
print(f" [{i+1}/{len(todo)}] rate={rate:.1f}/s eta={eta:.0f}s failed={failed}")
|
||||||
|
print(f" gender: {counts_g}")
|
||||||
|
print(f" race : {counts_r}")
|
||||||
|
|
||||||
|
# Write updated manifest atomically
|
||||||
|
tmp = manifest_path + ".tmp"
|
||||||
|
with open(tmp, "w") as f:
|
||||||
|
for r in rows:
|
||||||
|
f.write(json.dumps(r) + "\n")
|
||||||
|
os.replace(tmp, manifest_path)
|
||||||
|
|
||||||
|
final_g = {}
|
||||||
|
final_r = {}
|
||||||
|
excluded = 0
|
||||||
|
age_hist = {"<18": 0, "18-22": 0, "22-30": 0, "30-40": 0, "40-50": 0, "50-60": 0, "60+": 0, "unknown": 0}
|
||||||
|
for r in rows:
|
||||||
|
if r.get("excluded"):
|
||||||
|
excluded += 1
|
||||||
|
continue
|
||||||
|
final_g[r.get("gender") or "untagged"] = final_g.get(r.get("gender") or "untagged", 0) + 1
|
||||||
|
final_r[r.get("race") or "untagged"] = final_r.get(r.get("race") or "untagged", 0) + 1
|
||||||
|
a = r.get("age") or 0
|
||||||
|
if a == 0: age_hist["unknown"] += 1
|
||||||
|
elif a < 18: age_hist["<18"] += 1
|
||||||
|
elif a < 22: age_hist["18-22"] += 1
|
||||||
|
elif a < 30: age_hist["22-30"] += 1
|
||||||
|
elif a < 40: age_hist["30-40"] += 1
|
||||||
|
elif a < 50: age_hist["40-50"] += 1
|
||||||
|
elif a < 60: age_hist["50-60"] += 1
|
||||||
|
else: age_hist["60+"] += 1
|
||||||
|
print(f"\nDone. {len(rows)} rows, {excluded} excluded as <{args.min_age}, {failed} tag errors, {time.time()-t0:.1f}s")
|
||||||
|
print(f" final gender: {final_g}")
|
||||||
|
print(f" final race : {final_r}")
|
||||||
|
print(f" age dist : {age_hist}")
|
||||||
|
print(f"\nNext: poke /headshots/__reload to refresh the in-memory pool.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
x
Reference in New Issue
Block a user