a81291e38c
Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61 lines
2.4 KiB
Bash
61 lines
2.4 KiB
Bash
#!/usr/bin/env bash
|
|
# lib/env.sh — proof harness environment.
|
|
#
|
|
# Sourced once by run_proof.sh and by every case script. Establishes:
|
|
# - service URLs (gateway and direct ports)
|
|
# - report directory paths
|
|
# - run context (git SHA, hostname, timestamp)
|
|
# - mode (contract|integration|performance)
|
|
#
|
|
# Cases read from these vars; never re-set them.
|
|
|
|
# Repo root — every path the harness emits is anchored here so report
|
|
# JSON is portable across reviewer machines.
|
|
PROOF_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
|
export PROOF_REPO_ROOT
|
|
|
|
# Service endpoints. Match internal/shared/config.go defaults; every
|
|
# binary binds 127.0.0.1 per the audit's R-007 mitigation.
|
|
export PROOF_GATEWAY_URL="${PROOF_GATEWAY_URL:-http://127.0.0.1:3110}"
|
|
export PROOF_STORAGED_URL="${PROOF_STORAGED_URL:-http://127.0.0.1:3211}"
|
|
export PROOF_CATALOGD_URL="${PROOF_CATALOGD_URL:-http://127.0.0.1:3212}"
|
|
export PROOF_INGESTD_URL="${PROOF_INGESTD_URL:-http://127.0.0.1:3213}"
|
|
export PROOF_QUERYD_URL="${PROOF_QUERYD_URL:-http://127.0.0.1:3214}"
|
|
export PROOF_VECTORD_URL="${PROOF_VECTORD_URL:-http://127.0.0.1:3215}"
|
|
export PROOF_EMBEDD_URL="${PROOF_EMBEDD_URL:-http://127.0.0.1:3216}"
|
|
|
|
# Mode + report directory — set by run_proof.sh before sourcing cases.
|
|
# Defaulted here so cases sourced standalone for debug still work.
|
|
export PROOF_MODE="${PROOF_MODE:-contract}"
|
|
|
|
if [ -z "${PROOF_REPORT_DIR:-}" ]; then
|
|
ts="$(date -u +%Y%m%d-%H%M%SZ)"
|
|
export PROOF_REPORT_DIR="${PROOF_REPO_ROOT}/tests/proof/reports/proof-${ts}"
|
|
fi
|
|
|
|
mkdir -p \
|
|
"${PROOF_REPORT_DIR}/raw/http" \
|
|
"${PROOF_REPORT_DIR}/raw/logs" \
|
|
"${PROOF_REPORT_DIR}/raw/outputs" \
|
|
"${PROOF_REPORT_DIR}/raw/metrics" \
|
|
"${PROOF_REPORT_DIR}/raw/cases"
|
|
|
|
# Run context — captured once per run by run_proof.sh, but recomputed
|
|
# here as fallback if a case is invoked standalone.
|
|
if [ ! -f "${PROOF_REPORT_DIR}/raw/context.json" ]; then
|
|
git_sha="$(cd "$PROOF_REPO_ROOT" && git rev-parse HEAD 2>/dev/null || echo unknown)"
|
|
git_dirty="$(cd "$PROOF_REPO_ROOT" && [ -n "$(git status --porcelain 2>/dev/null)" ] && echo true || echo false)"
|
|
cat > "${PROOF_REPORT_DIR}/raw/context.json" <<JSON
|
|
{
|
|
"git_sha": "${git_sha}",
|
|
"git_dirty": ${git_dirty},
|
|
"hostname": "$(hostname)",
|
|
"timestamp_utc": "$(date -u -Iseconds)",
|
|
"mode": "${PROOF_MODE}",
|
|
"harness_version": "1"
|
|
}
|
|
JSON
|
|
fi
|
|
|
|
export PROOF_GIT_SHA="$(cd "$PROOF_REPO_ROOT" && git rev-parse HEAD 2>/dev/null || echo unknown)"
|