3263254f1c
Stress-tests the role gate with 40 queries (10 fill_events rows × 4
styles): need, client_first, looking, shorthand. Each row's role +
client + city stays the same; only the surface phrasing changes.
real_003 (original extractor) confirmed the shorthand-vs-shorthand
failure mode: CNC Operator shorthand recording leaked w-2404 onto
Forklift Operator shorthand query within the same Beacon Freight
Detroit cluster. Both record + query had empty role (extractor
returns "" for shorthand because there's no separator between role
and city), gate disabled, distance check passed, bleed fired.
Fix: extended extractRoleFromNeed to handle client_first
("{client} needs N {role} in...") and looking ("Looking for N
{role} at...") patterns. Shorthand left intentionally unmatched —
"Forklift Operator Detroit" is shape-indistinguishable from
"Forklift" + "Operator Detroit" without an LLM extractor or known-
cities lookup.
real_003b (extended extractor) verifies bleed closed across all 4
styles for this dataset. Forklift Operator queries keep w-2136 (the
cold-pass-correct match) regardless of which style the query came
in. Same-role boosts now fire correctly across styles — a CNC
Operator recording made in `looking` style boosts the CNC need-form
query.
scripts/cutover/gen_real_queries.go: added -styles flag with values
need|client_first|looking|shorthand|all (default need preserves
real_001/002 behavior). Tests/reality/real_coord_queries_v2.txt is
the 40-query stress file.
scripts/playbook_lift/main_test.go: 10 sub-tests lock the four
documented patterns + shorthand limitation + lift-suite-style
queries (no clean role, returns empty as expected).
Aggregate metrics:
- real_003 (original): disc=7, lift=7, boost=14, meanΔ=-0.108
- real_003b (extended): disc=11, lift=10, boost=31, meanΔ=-0.202
The growth reflects more LEGITIMATE same-role same-cluster transfer
firing across styles, not bleed (verified by per-cluster bleed
table — Forklift Operator queries unchanged across all 4 styles).
Known limitation documented in real_003_findings.md: same-cluster,
same-role queries in shorthand still embed close enough that a
shorthand recording could bleed onto a different-role shorthand
query if both record + query strip role. Closing this requires
LLM extraction or known-cities lookup at record + query time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
77 lines
2.3 KiB
Go
77 lines
2.3 KiB
Go
package main
|
|
|
|
import "testing"
|
|
|
|
// TestExtractRoleFromNeed locks the four query-shape patterns documented
|
|
// in real_003_findings.md so a future change to the regex can't silently
|
|
// drop coverage of any production-shape style. Real_001 used `need`-only;
|
|
// real_003 confirmed `shorthand` cross-role bleed; the extended
|
|
// extractor in real_003b covers `client_first` + `looking` and leaves
|
|
// `shorthand` as a known limitation (no separator between role and city).
|
|
func TestExtractRoleFromNeed(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
query string
|
|
want string
|
|
}{
|
|
{
|
|
"need style — original real_001 form",
|
|
"Need 1 Forklift Operator in Detroit MI starting at 15:00 for Beacon Freight",
|
|
"Forklift Operator",
|
|
},
|
|
{
|
|
"need with deadline trailer",
|
|
"Need 4 Pickers in Detroit MI starting at 13:30 for Beacon Freight, deadline 2026-05-28",
|
|
"Pickers",
|
|
},
|
|
{
|
|
"client_first style — added in real_003b",
|
|
"Beacon Freight needs 1 Forklift Operator in Detroit MI at 15:00",
|
|
"Forklift Operator",
|
|
},
|
|
{
|
|
"client_first with multi-word client",
|
|
"Parallel Machining needs 5 Warehouse Associates in Kansas City MO at 09:00",
|
|
"Warehouse Associates",
|
|
},
|
|
{
|
|
"looking style — added in real_003b",
|
|
"Looking for 1 Forklift Operator at Beacon Freight in Detroit MI for 15:00 shift",
|
|
"Forklift Operator",
|
|
},
|
|
{
|
|
"looking with multi-word role + 4-digit count",
|
|
"Looking for 1234 Senior Production Supervisors at Heritage Foods in Flint MI for 08:30 shift",
|
|
"Senior Production Supervisors",
|
|
},
|
|
{
|
|
"shorthand — known limitation, returns empty",
|
|
"1 Forklift Operator Detroit MI 15:00 Beacon Freight",
|
|
"",
|
|
},
|
|
{
|
|
"shorthand multi-word city — also empty",
|
|
"5 Warehouse Associates Kansas City MO 09:00 Parallel Machining",
|
|
"",
|
|
},
|
|
{
|
|
"lift-suite multi-constraint — no clean role, returns empty",
|
|
"Forklift operator with OSHA-30, warehouse experience, day shift availability",
|
|
"",
|
|
},
|
|
{
|
|
"OOD honesty signal — lift-suite, returns empty",
|
|
"Dental hygienist with three years experience, Indianapolis area",
|
|
"",
|
|
},
|
|
}
|
|
for _, c := range cases {
|
|
t.Run(c.name, func(t *testing.T) {
|
|
got := extractRoleFromNeed(c.query)
|
|
if got != c.want {
|
|
t.Errorf("extractRoleFromNeed(%q) = %q, want %q", c.query, got, c.want)
|
|
}
|
|
})
|
|
}
|
|
}
|