profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
golangLAKEHOUSE/reports/reality-tests/playbook_lift_real_004.md
root 0331288641 playbook_lift: LLM-based role extractor closes shorthand bleed (real_004) 
real_003 left a known-weak hole: shorthand-style queries
("{count} {role} {city} {state} ...") have no separator between
role and city, so a regex can't reliably extract — leaving the
cross-role gate disabled when both record AND query are shorthand.

This commit adds a roleExtractor with regex-first + LLM fallback:

- Regex first (fast, deterministic) — handles need + client_first +
  looking from real_003b. ~75% of styles, no LLM cost paid.
- LLM fallback when regex returns empty AND model is configured —
  Ollama-shape /api/chat with format=json, schema-tight prompt,
  temperature 0. ~1-3s on local qwen2.5.
- Per-process cache — paraphrase + rejudge passes reuse the same
  query 4× per run; cache prevents 4× LLM cost.
- Off-by-default — opt-in via -llm-role-extract flag (CLI) and
  LLM_ROLE_EXTRACT=1 env var (harness wrapper). real_003b shipping
  config unchanged unless explicitly enabled.

8 new tests in scripts/playbook_lift/main_test.go:
- TestRoleExtractor_RegexFirst: LLM not called when regex matches
- TestRoleExtractor_LLMFallback: shorthand goes to LLM
- TestRoleExtractor_LLMOffLeavesEmpty: opt-in default preserved
- TestRoleExtractor_Cache: 3 calls = 1 LLM hit
- TestRoleExtractor_NilSafe: nil receiver runs regex only
- TestExtractRoleViaLLM_HTTPError + _BadJSON: failure paths
- TestRoleExtractor_ClosesCrossRoleShorthandBleed: synthetic
  witness for the real_003 scenario — both record + query are
  shorthand, regex returns "" for both, LLM produces DIFFERENT
  role tokens for CNC vs Forklift, so matrix gate's cross-role
  rejection (locked separately in
  TestInjectPlaybookMisses_RoleGateRejectsCrossRole) fires
  correctly. This is the load-bearing verification.

Reality test real_004 ran the same 40-query stress as real_003 with
LLM extraction on. Cross-style same-role boosts fired correctly
across all 4 styles for Loaders + Packers + Shipping Clerk clusters
(including shorthand → other-style transfer). No cross-role bleed
observed. The reality test alone can't be a clean "with vs without"
comparison (HNSW build is non-deterministic across runs, and
real_004 stochastics didn't trigger a shorthand recording at all),
which is why the unit-test witness exists.

Production note (in real_004_findings.md): LLM extraction is for
reality-test coverage of arbitrary query shapes. Production should
extract role at INGEST time (when the inbox parser already runs an
LLM) and pass already-resolved role through requests — same shape
as multi_coord_stress's existing Demand{Role: ...} model. The hot
path should never need the harness extractor's per-query LLM cost.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 22:51:27 -05:00

117 lines
7.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Playbook-Lift Reality Test — Run real_004
**Generated:** 2026-05-01T03:48:39.932829485Z
**Judge:** `qwen2.5:latest` (Ollama, resolved from config [models].local_judge)
**Corpora:** `workers,ethereal_workers`
**Workers limit:** 5000
**Queries:** `tests/reality/real_coord_queries_v2.txt` (40 executed)
**K per pass:** 10
**Paraphrase pass:** disabled
**Re-judge pass:** disabled
**Evidence:** `reports/reality-tests/playbook_lift_real_004.json`
---
## Headline
| Metric | Value |
|---|---:|
| Total queries run | 40 |
| Cold-pass discoveries (judge-best ≠ top-1) | 5 |
| Warm-pass lifts (recorded playbook → top-1) | 4 |
| No change (judge-best already top-1, no playbook needed) | 36 |
| Playbook boosts triggered (warm pass) | 16 |
| Mean Δ top-1 distance (warm − cold) | -0.11025716 |
**Verbatim lift rate:** 4 of 5 discoveries became top-1 after warm pass.
---
## Per-query results
| # | Query | Cold top-1 | Cold judge-best (rank/rating) | Recorded? | Warm top-1 | Judge-best warm rank | Lift |
|---|---|---|---|---|---|---|---|
| 1 | Need 5 Warehouse Associates in Kansas City MO starting at 09 | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 2 | Parallel Machining needs 5 Warehouse Associates in Kansas Ci | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 3 | Looking for 5 Warehouse Associates at Parallel Machining in | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 4 | 5 Warehouse Associates Kansas City MO 09:00 Parallel Machini | w-3288 | 0/4 | — | w-3288 | 0 | no |
| 5 | Need 1 Forklift Operator in Detroit MI starting at 15:00 for | w-2136 | 0/5 | — | w-2136 | 0 | no |
| 6 | Beacon Freight needs 1 Forklift Operator in Detroit MI at 15 | e-6723 | 2/2 | — | e-6723 | 2 | no |
| 7 | Looking for 1 Forklift Operator at Beacon Freight in Detroit | w-4766 | 0/5 | — | w-4766 | 0 | no |
| 8 | 1 Forklift Operator Detroit MI 15:00 Beacon Freight | e-5818 | 0/4 | — | e-5818 | 0 | no |
| 9 | Need 4 Loaders in Indianapolis IN starting at 12:00 for Midw | e-9877 | 0/4 | — | w-398 | 2 | no |
| 10 | Midway Distribution needs 4 Loaders in Indianapolis IN at 12 | e-4537 | 4/5 | ✓ w-2148 | w-398 | 1 | no |
| 11 | Looking for 4 Loaders at Midway Distribution in Indianapolis | w-962 | 6/4 | ✓ w-398 | w-398 | 0 | **YES** |
| 12 | 4 Loaders Indianapolis IN 12:00 Midway Distribution | e-9877 | 0/4 | — | w-2148 | 2 | no |
| 13 | Need 3 Warehouse Associates in Fort Wayne IN starting at 17: | w-2857 | 1/4 | ✓ w-1063 | w-1063 | 0 | **YES** |
| 14 | Cornerstone Fabrication needs 3 Warehouse Associates in Fort | w-4703 | 0/2 | — | w-1063 | 1 | no |
| 15 | Looking for 3 Warehouse Associates at Cornerstone Fabricatio | w-1784 | 0/3 | — | w-1063 | 1 | no |
| 16 | 3 Warehouse Associates Fort Wayne IN 17:30 Cornerstone Fabri | w-2754 | 2/3 | — | w-1063 | 3 | no |
| 17 | Need 4 Pickers in Detroit MI starting at 13:30 for Beacon Fr | e-2247 | 1/2 | — | e-2247 | 1 | no |
| 18 | Beacon Freight needs 4 Pickers in Detroit MI at 13:30 | e-2247 | 2/2 | — | e-2247 | 2 | no |
| 19 | Looking for 4 Pickers at Beacon Freight in Detroit MI for 13 | e-2247 | 1/2 | — | e-2247 | 1 | no |
| 20 | 4 Pickers Detroit MI 13:30 Beacon Freight | e-2247 | 0/3 | — | e-2247 | 0 | no |
| 21 | Need 2 Packers in Joliet IL starting at 09:30 for Parallel M | e-846 | 2/3 | — | e-2120 | 0 | no |
| 22 | Parallel Machining needs 2 Packers in Joliet IL at 09:30 | e-846 | 2/4 | ✓ e-2120 | e-2120 | 0 | **YES** |
| 23 | Looking for 2 Packers at Parallel Machining in Joliet IL for | e-846 | 1/2 | — | e-2120 | 2 | no |
| 24 | 2 Packers Joliet IL 09:30 Parallel Machining | e-846 | 2/3 | — | e-2120 | 0 | no |
| 25 | Need 3 Assemblers in Flint MI starting at 08:30 for Heritage | w-4124 | 9/3 | — | w-4124 | 9 | no |
| 26 | Heritage Foods needs 3 Assemblers in Flint MI at 08:30 | w-4124 | 3/3 | — | w-4124 | 3 | no |
| 27 | Looking for 3 Assemblers at Heritage Foods in Flint MI for 0 | w-4124 | 2/2 | — | w-4124 | 2 | no |
| 28 | 3 Assemblers Flint MI 08:30 Heritage Foods | w-4124 | 3/3 | — | w-4124 | 3 | no |
| 29 | Need 3 Packers in Flint MI starting at 12:30 for Parallel Ma | e-6019 | 8/2 | — | e-6019 | 8 | no |
| 30 | Parallel Machining needs 3 Packers in Flint MI at 12:30 | e-6019 | 9/2 | — | e-6019 | 9 | no |
| 31 | Looking for 3 Packers at Parallel Machining in Flint MI for | e-6019 | 0/1 | — | e-6019 | 0 | no |
| 32 | 3 Packers Flint MI 12:30 Parallel Machining | e-2006 | 0/3 | — | e-2006 | 0 | no |
| 33 | Need 1 Shipping Clerk in Flint MI starting at 17:00 for Pion | w-3988 | 2/4 | ✓ w-1367 | w-1367 | 0 | **YES** |
| 34 | Pioneer Assembly needs 1 Shipping Clerk in Flint MI at 17:00 | w-3988 | 1/3 | — | w-1367 | 2 | no |
| 35 | Looking for 1 Shipping Clerk at Pioneer Assembly in Flint MI | w-3988 | 0/2 | — | w-1367 | 1 | no |
| 36 | 1 Shipping Clerk Flint MI 17:00 Pioneer Assembly | e-4849 | 0/2 | — | w-1367 | 1 | no |
| 37 | Need 1 CNC Operator in Detroit MI starting at 17:30 for Beac | e-489 | 5/3 | — | e-489 | 5 | no |
| 38 | Beacon Freight needs 1 CNC Operator in Detroit MI at 17:30 | e-489 | 0/2 | — | e-489 | 0 | no |
| 39 | Looking for 1 CNC Operator at Beacon Freight in Detroit MI f | e-489 | 0/2 | — | e-489 | 0 | no |
| 40 | 1 CNC Operator Detroit MI 17:30 Beacon Freight | e-489 | 3/3 | — | e-489 | 3 | no |
---
## Honesty caveats
1. **Judge IS the ground truth proxy.** Without human-labeled relevance, the LLM
judge's verdict is what defines "best." If `` rates badly,
the lift number is meaningless. To validate the judge itself, sample 5–10
verdicts manually and check agreement.
2. **Score-1.0 boost = distance halved.** Playbook math is
`distance' = distance × (1 - 0.5 × score)`. Lift requires the judge-best
result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise
even halving doesn't promote it. Tight clusters → little visible lift.
3. **Verbatim vs paraphrase.** The verbatim lift rate (above) is the cheap
case — same query, recorded playbook, expected boost. The paraphrase
pass (when enabled) is the actual learning property: similar-but-different
queries hitting a recorded playbook. Compare verbatim and paraphrase
lift rates — paraphrase should be lower (semantic-distance gates some
playbook hits) but non-zero is the meaningful signal.
4. **Multi-corpus skew.** Default corpora=`workers,ethereal_workers` — if all judge-best
results land in one corpus, the matrix layer's purpose isn't being tested.
Check per-corpus distribution in the JSON.
5. **Judge resolution.** This run used `qwen2.5:latest` from
config [models].local_judge.
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
6. **Paraphrase generation also uses the judge.** The same model that rates
relevance also rephrases queries. A judge that's bad at rating staffing
queries is probably also bad at rephrasing them. Worth sanity-checking
a sample of `paraphrase_query` values in the JSON before trusting the
paraphrase lift number.
## Next moves
- If lift rate ≥ 50% of discoveries: matrix layer + playbook is doing real
work. Move to paraphrase queries + tag-based boost (currently ignored).
- If lift rate < 20%: investigate why — judge variance, distance gap too
wide, or playbook math too gentle. The score=1.0 / 0.5× formula may need
retuning.
- If discovery rate (cold judge-best ≠ top-1) is itself low: cosine is
already close to optimal on this query distribution. Either the corpus
is too narrow or the queries are too easy.
Reference in New Issue View Git Blame Copy Permalink