Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
112 lines
3.8 KiB
Makefile
112 lines
3.8 KiB
Makefile
# golangLAKEHOUSE — task runner.
|
|
#
|
|
# Sprint 0 acceptance gate (R-004): smokes are no longer documentation
|
|
# only — `just verify` is the single command that runs vet + tests +
|
|
# the 9 smokes. The pre-push hook calls this; CI calls this; reviewers
|
|
# call this. One source of truth.
|
|
#
|
|
# Usage:
|
|
# just # alias for `just --list`
|
|
# just verify # vet + test + all 9 smokes (full gate)
|
|
# just smoke <day> # single smoke (d1..d6, g1, g1p, g2)
|
|
# just smoke-all # all 9 smokes only
|
|
# just doctor # dependency probe
|
|
# just fmt / vet / test / build
|
|
|
|
# Go lives at /usr/local/go/bin per ADR-001 §1.x; prepend so every
|
|
# recipe sees it without depending on the parent shell's PATH.
|
|
export PATH := "/usr/local/go/bin:" + env('PATH', '')
|
|
|
|
# Default recipe shows the menu so `just` alone is a discoverable entry point.
|
|
default:
|
|
@just --list
|
|
|
|
# Full Sprint 0 gate: vet + tests + 9 smokes. Pre-push hook calls this.
|
|
verify: vet test smoke-all
|
|
@echo ""
|
|
@echo "[verify] PASS — go vet + go test + 9 smokes all green"
|
|
|
|
# Static analysis. Runs first so we fail fast on syntax / shape issues.
|
|
vet:
|
|
@echo "[vet] go vet ./..."
|
|
@go vet ./...
|
|
|
|
# Go unit tests, short mode. Excludes hardware-in-the-loop tags.
|
|
test:
|
|
@echo "[test] go test -short -count=1 ./..."
|
|
@go test -short -count=1 ./...
|
|
|
|
# Format Go source. Idempotent; CI can run with --check via `just fmt-check`.
|
|
fmt:
|
|
@gofmt -w cmd internal scripts
|
|
|
|
# Verify formatting without modifying. Non-zero exit means run `just fmt`.
|
|
fmt-check:
|
|
@diff -u <(echo -n) <(gofmt -d cmd internal scripts)
|
|
|
|
# Build every binary into bin/. Mirrors what each smoke does internally.
|
|
build:
|
|
@echo "[build] go build -o bin/ ./cmd/..."
|
|
@go build -o bin/ ./cmd/...
|
|
|
|
# Single smoke. Day is the suffix before _smoke.sh — d1, d2, …, g2.
|
|
smoke day:
|
|
@bash scripts/{{day}}_smoke.sh
|
|
|
|
# All 9 smokes in dependency order. Halts on first failure.
|
|
smoke-all:
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
for day in d1 d2 d3 d4 d5 d6 g1 g1p g2; do
|
|
printf "[smoke-all] %s ... " "$day"
|
|
SECONDS=0
|
|
if bash "scripts/${day}_smoke.sh" >/tmp/smoke_${day}.log 2>&1; then
|
|
printf "PASS (%ss)\n" "$SECONDS"
|
|
else
|
|
printf "FAIL (%ss)\n" "$SECONDS"
|
|
echo ""
|
|
echo " last 20 lines of /tmp/smoke_${day}.log:"
|
|
tail -20 "/tmp/smoke_${day}.log" | sed 's/^/ /'
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
# Dependency probe. Add --json for machine-readable output.
|
|
doctor *args:
|
|
@bash scripts/doctor.sh {{args}}
|
|
|
|
# Proof harness — claims-verification tier above the smoke chain.
|
|
# See tests/proof/README.md and docs/TEST_PROOF_SCOPE.md.
|
|
# just proof contract fast: APIs + status codes + dim/nonempty
|
|
# just proof integration full: CSV→Parquet→SQL, text→vector→search
|
|
# just proof performance measurements; runs only after contract+integration
|
|
proof mode *flags:
|
|
@bash tests/proof/run_proof.sh --mode {{mode}} {{flags}}
|
|
|
|
# Install pre-push hook so `git push` runs `just verify` first.
|
|
install-hooks:
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
HOOK=".git/hooks/pre-push"
|
|
cat > "$HOOK" <<'HOOK'
|
|
#!/usr/bin/env bash
|
|
# golangLAKEHOUSE pre-push hook (managed by `just install-hooks`).
|
|
# Runs the Sprint 0 gate before letting commits leave this machine.
|
|
set -e
|
|
cd "$(git rev-parse --show-toplevel)"
|
|
echo "[pre-push] running just verify ..."
|
|
if ! just verify; then
|
|
echo ""
|
|
echo "[pre-push] FAIL — push aborted. Fix the gate or use --no-verify (NOT recommended)."
|
|
exit 1
|
|
fi
|
|
HOOK
|
|
chmod +x "$HOOK"
|
|
echo "[install-hooks] $HOOK installed and executable"
|
|
|
|
# Clean built binaries + smoke logs. Does NOT touch reports/ or data/.
|
|
clean:
|
|
@rm -rf bin/
|
|
@rm -f /tmp/smoke_*.log
|
|
@echo "[clean] bin/ removed, smoke logs cleared"
|