3dd7d9fe30
First reality test driver. Two-pass design:
- Pass 1 (cold): matrix.search use_playbook=false → small-model judge
rates top-K → record playbook entry pointing at the highest-rated
result (which may NOT be top-1 by distance — that's the discovery).
- Pass 2 (warm): same queries with use_playbook=true → measure
ranking shift. Lift = real if recorded answer becomes top-1.
Files:
- scripts/playbook_lift/main.go driver (391 LoC)
- scripts/playbook_lift.sh stack-bring-up + report gen
- tests/reality/playbook_lift_queries.txt query corpus (5 placeholders;
J writes real 20+)
- reports/reality-tests/README.md framework + interpretation
- .gitignore track reports/reality-tests/
but ignore per-run JSON evidence
This answers the gate from project_small_model_pipeline_vision.md:
"the playbook + matrix indexer must give the results we're looking
for." Without ground-truth labels, the LLM judge is the proxy — the
same small-model thesis applied to evaluation. Honest about that
limitation in the generated reports.
Driver compiles clean; full run requires Ollama + workers/candidates
ingest. Skips cleanly if Ollama absent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
60 lines
1.4 KiB
Plaintext
60 lines
1.4 KiB
Plaintext
# Go
|
|
*.exe
|
|
*.exe~
|
|
*.dll
|
|
*.so
|
|
*.dylib
|
|
*.test
|
|
*.out
|
|
go.work
|
|
go.work.sum
|
|
vendor/
|
|
|
|
# Build artifacts
|
|
/bin/
|
|
/dist/
|
|
|
|
# Editor / OS
|
|
.DS_Store
|
|
.idea/
|
|
.vscode/
|
|
*.swp
|
|
*~
|
|
|
|
# Local data — these directories follow the Rust lakehouse pattern;
|
|
# regenerated by services on demand. Do not commit runtime artifacts.
|
|
/data/_auditor/
|
|
/data/_kb/
|
|
/data/_pathway_memory/
|
|
/data/_errors/
|
|
/data/_imagecache/
|
|
/data/datasets/
|
|
/data/vectors/
|
|
/data/headshots/
|
|
/data/lance/
|
|
/exports/
|
|
/logs/
|
|
# /reports/ holds runtime artifacts by default (matches Rust lakehouse
|
|
# convention) — but reports/scrum/ is intentional audit documentation.
|
|
# Use /reports/* + un-ignore so git can traverse into reports/.
|
|
/reports/*
|
|
!/reports/scrum/
|
|
!/reports/reality-tests/
|
|
# Inside the audit directory, the per-run _evidence/ dump (smoke logs,
|
|
# command output) IS runtime — track the dir, ignore its contents.
|
|
/reports/scrum/_evidence/*
|
|
!/reports/scrum/_evidence/.gitkeep
|
|
# Reality-test JSON evidence is runtime — track the dir + MD reports
|
|
# (committed deliberately as outcome record), ignore per-run JSON.
|
|
/reports/reality-tests/*.json
|
|
|
|
# Proof harness runtime output — same pattern as reports/scrum/_evidence.
|
|
# Track the directory but ignore per-run subdirs.
|
|
/tests/proof/reports/*
|
|
!/tests/proof/reports/.gitkeep
|
|
|
|
# Secrets — never commit. Resolved via SecretsProvider per ADR-001 §1.x.
|
|
*.env
|
|
secrets.toml
|
|
secrets-go.toml
|