7f2f112e6a
First retrieval probe with non-synthetic query distribution. Pulls
N rows from /home/profit/lakehouse/data/datasets/fill_events.parquet
(real-shape demand data) and translates each to the natural language
a coordinator would type: "Need {count} {role}s in {city} {state}
starting at {at} for {client}".
Headline: 8/10 cold-pass top-1 = judge-best on real distribution.
Substrate works on queries it was never trained for. v2-moe + workers
corpus carry the load.
Surfaced finding (the real value of running this): same-client+city
queries cluster, and Shape A's distance boost bleeds across roles
within the cluster. Q#2 (Forklift @ Beacon Freight Detroit) records
e-6193 in the playbook corpus. Q#5 (Pickers same client+city) and
Q#10 (CNC Operator same client+city) inherit e-6193 at warm top-1
even though:
- Neither query has its own recorded playbook.
- Neither warm pass triggers a Shape B inject (boosted=0).
- The roles are different staffing categories.
Q#10 specifically demoted the cold-pass-correct w-3759 (judge rating
4 at rank 0) for a worker who was approved by the judge for a
different role on a different query.
Why the lift suite missed it: synthetic queries use 7 disjoint
scenario buckets (forklift+OSHA+WI / CDL+IL / etc.). Real demand
clusters on (client, city). The cluster doesn't exist in the
synthetic distribution.
Why the judge gate doesn't catch it: the gate (5a3364f) is
per-injection at record time. After approval the worker rides Shape A
distance boosts on all later same-cluster queries with no second
gate call.
Becomes new OPEN #1. Fix candidate: role-scoped playbook corpus
metadata + Shape A boost gate on role match. Cheap; doesn't need
new judge calls.
Files:
- scripts/cutover/gen_real_queries.go: parquet → coordinator NL
- tests/reality/real_coord_queries.txt: 10 generated queries
- reports/reality-tests/playbook_lift_real_001.md: harness output
- reports/reality-tests/real_001_findings.md: the reading
Repro:
go run scripts/cutover/gen_real_queries.go -limit 10 > tests/reality/real_coord_queries.txt
QUERIES_FILE=tests/reality/real_coord_queries.txt RUN_ID=real_001 \
WITH_PARAPHRASE=0 WITH_REJUDGE=0 ./scripts/playbook_lift.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
87 lines
4.3 KiB
Markdown
87 lines
4.3 KiB
Markdown
# Playbook-Lift Reality Test — Run real_001
|
||
|
||
**Generated:** 2026-05-01T01:15:54.879146128Z
|
||
**Judge:** `qwen2.5:latest` (Ollama, resolved from config [models].local_judge)
|
||
**Corpora:** `workers,ethereal_workers`
|
||
**Workers limit:** 5000
|
||
**Queries:** `tests/reality/real_coord_queries.txt` (10 executed)
|
||
**K per pass:** 10
|
||
**Paraphrase pass:** disabled
|
||
**Re-judge pass:** disabled
|
||
**Evidence:** `reports/reality-tests/playbook_lift_real_001.json`
|
||
|
||
---
|
||
|
||
## Headline
|
||
|
||
| Metric | Value |
|
||
|---|---:|
|
||
| Total queries run | 10 |
|
||
| Cold-pass discoveries (judge-best ≠ top-1) | 2 |
|
||
| Warm-pass lifts (recorded playbook → top-1) | 2 |
|
||
| No change (judge-best already top-1, no playbook needed) | 8 |
|
||
| Playbook boosts triggered (warm pass) | 2 |
|
||
| Mean Δ top-1 distance (warm − cold) | -0.12680706 |
|
||
|
||
|
||
|
||
**Verbatim lift rate:** 2 of 2 discoveries became top-1 after warm pass.
|
||
|
||
---
|
||
|
||
## Per-query results
|
||
|
||
| # | Query | Cold top-1 | Cold judge-best (rank/rating) | Recorded? | Warm top-1 | Judge-best warm rank | Lift |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 1 | Need 5 Warehouse Associates in Kansas City MO starting at 09 | w-4781 | 0/4 | — | w-4781 | 0 | no |
|
||
| 2 | Need 1 Forklift Operator in Detroit MI starting at 15:00 for | e-5617 | 2/4 | ✓ e-6193 | e-6193 | 0 | **YES** |
|
||
| 3 | Need 4 Loaders in Indianapolis IN starting at 12:00 for Midw | e-9877 | 0/4 | — | e-9877 | 0 | no |
|
||
| 4 | Need 3 Warehouse Associates in Fort Wayne IN starting at 17: | w-3370 | 0/4 | — | w-3370 | 0 | no |
|
||
| 5 | Need 4 Pickers in Detroit MI starting at 13:30 for Beacon Fr | e-5620 | 1/2 | — | e-6193 | 2 | no |
|
||
| 6 | Need 2 Packers in Joliet IL starting at 09:30 for Parallel M | e-1746 | 0/2 | — | e-1746 | 0 | no |
|
||
| 7 | Need 3 Assemblers in Flint MI starting at 08:30 for Heritage | w-4124 | 0/2 | — | w-4124 | 0 | no |
|
||
| 8 | Need 3 Packers in Flint MI starting at 12:30 for Parallel Ma | e-7325 | 0/2 | — | e-7325 | 0 | no |
|
||
| 9 | Need 1 Shipping Clerk in Flint MI starting at 17:00 for Pion | w-3988 | 4/4 | ✓ w-1367 | w-1367 | 0 | **YES** |
|
||
| 10 | Need 1 CNC Operator in Detroit MI starting at 17:30 for Beac | w-3759 | 0/4 | — | e-6193 | 1 | no |
|
||
|
||
---
|
||
|
||
## Honesty caveats
|
||
|
||
1. **Judge IS the ground truth proxy.** Without human-labeled relevance, the LLM
|
||
judge's verdict is what defines "best." If `` rates badly,
|
||
the lift number is meaningless. To validate the judge itself, sample 5–10
|
||
verdicts manually and check agreement.
|
||
2. **Score-1.0 boost = distance halved.** Playbook math is
|
||
`distance' = distance × (1 - 0.5 × score)`. Lift requires the judge-best
|
||
result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise
|
||
even halving doesn't promote it. Tight clusters → little visible lift.
|
||
3. **Verbatim vs paraphrase.** The verbatim lift rate (above) is the cheap
|
||
case — same query, recorded playbook, expected boost. The paraphrase
|
||
pass (when enabled) is the actual learning property: similar-but-different
|
||
queries hitting a recorded playbook. Compare verbatim and paraphrase
|
||
lift rates — paraphrase should be lower (semantic-distance gates some
|
||
playbook hits) but non-zero is the meaningful signal.
|
||
4. **Multi-corpus skew.** Default corpora=`workers,ethereal_workers` — if all judge-best
|
||
results land in one corpus, the matrix layer's purpose isn't being tested.
|
||
Check per-corpus distribution in the JSON.
|
||
5. **Judge resolution.** This run used `qwen2.5:latest` from
|
||
config [models].local_judge.
|
||
Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
|
||
6. **Paraphrase generation also uses the judge.** The same model that rates
|
||
relevance also rephrases queries. A judge that's bad at rating staffing
|
||
queries is probably also bad at rephrasing them. Worth sanity-checking
|
||
a sample of `paraphrase_query` values in the JSON before trusting the
|
||
paraphrase lift number.
|
||
|
||
## Next moves
|
||
|
||
- If lift rate ≥ 50% of discoveries: matrix layer + playbook is doing real
|
||
work. Move to paraphrase queries + tag-based boost (currently ignored).
|
||
- If lift rate < 20%: investigate why — judge variance, distance gap too
|
||
wide, or playbook math too gentle. The score=1.0 / 0.5× formula may need
|
||
retuning.
|
||
- If discovery rate (cold judge-best ≠ top-1) is itself low: cosine is
|
||
already close to optimal on this query distribution. Either the corpus
|
||
is too narrow or the queries are too easy.
|