root 7f2f112e6a reality_test real_001: real-shape coordinator queries — surfaces cross-role bleed

First retrieval probe with non-synthetic query distribution. Pulls
N rows from /home/profit/lakehouse/data/datasets/fill_events.parquet
(real-shape demand data) and translates each to the natural language
a coordinator would type: "Need {count} {role}s in {city} {state}
starting at {at} for {client}".

Headline: 8/10 cold-pass top-1 = judge-best on real distribution.
Substrate works on queries it was never trained for. v2-moe + workers
corpus carry the load.

Surfaced finding (the real value of running this): same-client+city
queries cluster, and Shape A's distance boost bleeds across roles
within the cluster. Q#2 (Forklift @ Beacon Freight Detroit) records
e-6193 in the playbook corpus. Q#5 (Pickers same client+city) and
Q#10 (CNC Operator same client+city) inherit e-6193 at warm top-1
even though:
- Neither query has its own recorded playbook.
- Neither warm pass triggers a Shape B inject (boosted=0).
- The roles are different staffing categories.

Q#10 specifically demoted the cold-pass-correct w-3759 (judge rating
4 at rank 0) for a worker who was approved by the judge for a
different role on a different query.

Why the lift suite missed it: synthetic queries use 7 disjoint
scenario buckets (forklift+OSHA+WI / CDL+IL / etc.). Real demand
clusters on (client, city). The cluster doesn't exist in the
synthetic distribution.

Why the judge gate doesn't catch it: the gate (5a3364f) is
per-injection at record time. After approval the worker rides Shape A
distance boosts on all later same-cluster queries with no second
gate call.

Becomes new OPEN #1. Fix candidate: role-scoped playbook corpus
metadata + Shape A boost gate on role match. Cheap; doesn't need
new judge calls.

Files:
- scripts/cutover/gen_real_queries.go: parquet → coordinator NL
- tests/reality/real_coord_queries.txt: 10 generated queries
- reports/reality-tests/playbook_lift_real_001.md: harness output
- reports/reality-tests/real_001_findings.md: the reading

Repro:
  go run scripts/cutover/gen_real_queries.go -limit 10 > tests/reality/real_coord_queries.txt
  QUERIES_FILE=tests/reality/real_coord_queries.txt RUN_ID=real_001 \
    WITH_PARAPHRASE=0 WITH_REJUDGE=0 ./scripts/playbook_lift.sh

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-30 20:18:40 -05:00

4.3 KiB

Raw Permalink Blame History

Playbook-Lift Reality Test — Run real_001

Generated: 2026-05-01T01:15:54.879146128Z Judge: qwen2.5:latest (Ollama, resolved from config [models].local_judge) Corpora: workers,ethereal_workers Workers limit: 5000 Queries: tests/reality/real_coord_queries.txt (10 executed) K per pass: 10 Paraphrase pass: disabled Re-judge pass: disabled Evidence: reports/reality-tests/playbook_lift_real_001.json

Headline

Metric	Value
Total queries run	10
Cold-pass discoveries (judge-best ≠ top-1)	2
Warm-pass lifts (recorded playbook → top-1)	2
No change (judge-best already top-1, no playbook needed)	8
Playbook boosts triggered (warm pass)	2
Mean Δ top-1 distance (warm − cold)	-0.12680706

Verbatim lift rate: 2 of 2 discoveries became top-1 after warm pass.

Per-query results

#	Query	Cold top-1	Cold judge-best (rank/rating)	Recorded?	Warm top-1	Judge-best warm rank	Lift
1	Need 5 Warehouse Associates in Kansas City MO starting at 09	w-4781	0/4	—	w-4781	0	no
2	Need 1 Forklift Operator in Detroit MI starting at 15:00 for	e-5617	2/4	✓ e-6193	e-6193	0	YES
3	Need 4 Loaders in Indianapolis IN starting at 12:00 for Midw	e-9877	0/4	—	e-9877	0	no
4	Need 3 Warehouse Associates in Fort Wayne IN starting at 17:	w-3370	0/4	—	w-3370	0	no
5	Need 4 Pickers in Detroit MI starting at 13:30 for Beacon Fr	e-5620	1/2	—	e-6193	2	no
6	Need 2 Packers in Joliet IL starting at 09:30 for Parallel M	e-1746	0/2	—	e-1746	0	no
7	Need 3 Assemblers in Flint MI starting at 08:30 for Heritage	w-4124	0/2	—	w-4124	0	no
8	Need 3 Packers in Flint MI starting at 12:30 for Parallel Ma	e-7325	0/2	—	e-7325	0	no
9	Need 1 Shipping Clerk in Flint MI starting at 17:00 for Pion	w-3988	4/4	✓ w-1367	w-1367	0	YES
10	Need 1 CNC Operator in Detroit MI starting at 17:30 for Beac	w-3759	0/4	—	e-6193	1	no

Honesty caveats

Judge IS the ground truth proxy. Without human-labeled relevance, the LLM judge's verdict is what defines "best." If `` rates badly, the lift number is meaningless. To validate the judge itself, sample 5–10 verdicts manually and check agreement.
Score-1.0 boost = distance halved. Playbook math is distance' = distance × (1 - 0.5 × score). Lift requires the judge-best result's pre-boost distance to be ≤ 2× the cold top-1's distance, otherwise even halving doesn't promote it. Tight clusters → little visible lift.
Verbatim vs paraphrase. The verbatim lift rate (above) is the cheap case — same query, recorded playbook, expected boost. The paraphrase pass (when enabled) is the actual learning property: similar-but-different queries hitting a recorded playbook. Compare verbatim and paraphrase lift rates — paraphrase should be lower (semantic-distance gates some playbook hits) but non-zero is the meaningful signal.
Multi-corpus skew. Default corpora=workers,ethereal_workers — if all judge-best results land in one corpus, the matrix layer's purpose isn't being tested. Check per-corpus distribution in the JSON.
Judge resolution. This run used qwen2.5:latest from config [models].local_judge. Bumping the judge for run #N+1 means editing one line in lakehouse.toml.
Paraphrase generation also uses the judge. The same model that rates relevance also rephrases queries. A judge that's bad at rating staffing queries is probably also bad at rephrasing them. Worth sanity-checking a sample of paraphrase_query values in the JSON before trusting the paraphrase lift number.

Next moves

If lift rate ≥ 50% of discoveries: matrix layer + playbook is doing real work. Move to paraphrase queries + tag-based boost (currently ignored).
If lift rate < 20%: investigate why — judge variance, distance gap too wide, or playbook math too gentle. The score=1.0 / 0.5× formula may need retuning.
If discovery rate (cold judge-best ≠ top-1) is itself low: cosine is already close to optimal on this query distribution. Either the corpus is too narrow or the queries are too easy.

4.3 KiB Raw Permalink Blame History Unescape Escape