a81291e38c
Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
119 lines
4.5 KiB
Bash
119 lines
4.5 KiB
Bash
#!/usr/bin/env bash
|
|
# lib/assert.sh — assertions that record evidence per the spec.
|
|
#
|
|
# Each assertion appends one record to:
|
|
# $PROOF_REPORT_DIR/raw/cases/<case_id>.jsonl
|
|
#
|
|
# Each line is a self-describing JSON object — case ID, claim, expected,
|
|
# actual, result {pass|fail|skip}, optional evidence pointers. Cases
|
|
# call multiple assertions; run_proof.sh aggregates JSONL → summary.
|
|
#
|
|
# Functions:
|
|
# proof_assert_eq <case_id> <claim> <expected> <actual>
|
|
# proof_assert_ne <case_id> <claim> <not_expected> <actual>
|
|
# proof_assert_contains <case_id> <claim> <substring> <haystack>
|
|
# proof_assert_lt <case_id> <claim> <a> <b> # passes if a < b
|
|
# proof_assert_gt <case_id> <claim> <a> <b> # passes if a > b
|
|
# proof_assert_status <case_id> <claim> <expected_status> <probe_name>
|
|
# proof_assert_json_eq <case_id> <claim> <jq_path> <expected> <body_or_file>
|
|
# proof_skip <case_id> <claim> <reason>
|
|
#
|
|
# All return 0 (case scripts decide their own halt-on-fail policy).
|
|
|
|
_proof_record() {
|
|
local case_id="$1" claim="$2" result="$3" expected="$4" actual="$5" detail="$6"
|
|
local out="${PROOF_REPORT_DIR}/raw/cases/${case_id}.jsonl"
|
|
mkdir -p "$(dirname "$out")"
|
|
# JSON-escape the variable inputs.
|
|
local e_claim e_expected e_actual e_detail
|
|
e_claim=$(printf '%s' "$claim" | jq -Rs .)
|
|
e_expected=$(printf '%s' "$expected" | jq -Rs .)
|
|
e_actual=$(printf '%s' "$actual" | jq -Rs .)
|
|
e_detail=$(printf '%s' "$detail" | jq -Rs .)
|
|
local ts
|
|
ts=$(date -u -Iseconds)
|
|
cat >> "$out" <<JSON
|
|
{"case_id":"${case_id}","claim":${e_claim},"result":"${result}","expected":${e_expected},"actual":${e_actual},"detail":${e_detail},"timestamp":"${ts}","git_sha":"${PROOF_GIT_SHA}"}
|
|
JSON
|
|
}
|
|
|
|
proof_assert_eq() {
|
|
local case_id="$1" claim="$2" expected="$3" actual="$4"
|
|
if [ "$expected" = "$actual" ]; then
|
|
_proof_record "$case_id" "$claim" pass "$expected" "$actual" ""
|
|
return 0
|
|
fi
|
|
_proof_record "$case_id" "$claim" fail "$expected" "$actual" "values differ"
|
|
return 0
|
|
}
|
|
|
|
proof_assert_ne() {
|
|
local case_id="$1" claim="$2" not_expected="$3" actual="$4"
|
|
if [ "$not_expected" != "$actual" ]; then
|
|
_proof_record "$case_id" "$claim" pass "!= ${not_expected}" "$actual" ""
|
|
return 0
|
|
fi
|
|
_proof_record "$case_id" "$claim" fail "!= ${not_expected}" "$actual" "values matched (should differ)"
|
|
return 0
|
|
}
|
|
|
|
proof_assert_contains() {
|
|
local case_id="$1" claim="$2" substring="$3" haystack="$4"
|
|
if [[ "$haystack" == *"$substring"* ]]; then
|
|
_proof_record "$case_id" "$claim" pass "contains: ${substring}" "$haystack" ""
|
|
return 0
|
|
fi
|
|
_proof_record "$case_id" "$claim" fail "contains: ${substring}" "$haystack" "substring not found"
|
|
return 0
|
|
}
|
|
|
|
proof_assert_lt() {
|
|
local case_id="$1" claim="$2" a="$3" b="$4"
|
|
# awk handles ints + floats uniformly
|
|
if awk -v a="$a" -v b="$b" 'BEGIN{exit !(a < b)}'; then
|
|
_proof_record "$case_id" "$claim" pass "${a} < ${b}" "${a}" ""
|
|
return 0
|
|
fi
|
|
_proof_record "$case_id" "$claim" fail "${a} < ${b}" "${a}" "${a} is not less than ${b}"
|
|
return 0
|
|
}
|
|
|
|
proof_assert_gt() {
|
|
local case_id="$1" claim="$2" a="$3" b="$4"
|
|
if awk -v a="$a" -v b="$b" 'BEGIN{exit !(a > b)}'; then
|
|
_proof_record "$case_id" "$claim" pass "${a} > ${b}" "${a}" ""
|
|
return 0
|
|
fi
|
|
_proof_record "$case_id" "$claim" fail "${a} > ${b}" "${a}" "${a} is not greater than ${b}"
|
|
return 0
|
|
}
|
|
|
|
# proof_assert_status compares the status from a previously-recorded
|
|
# probe against an expected value. Probe must have run via lib/http.sh.
|
|
proof_assert_status() {
|
|
local case_id="$1" claim="$2" expected="$3" probe_name="$4"
|
|
local actual
|
|
actual=$(proof_status_of "$case_id" "$probe_name" 2>/dev/null || echo missing)
|
|
proof_assert_eq "$case_id" "$claim" "$expected" "$actual"
|
|
}
|
|
|
|
# proof_assert_json_eq: jq-based equality on response body or a file.
|
|
# body_or_file: if starts with @, read from file; otherwise treat as
|
|
# literal JSON string.
|
|
proof_assert_json_eq() {
|
|
local case_id="$1" claim="$2" jq_path="$3" expected="$4" source="$5"
|
|
local actual
|
|
if [[ "$source" == @* ]]; then
|
|
actual=$(jq -r "$jq_path" "${source#@}" 2>/dev/null || echo "<jq error>")
|
|
else
|
|
actual=$(printf '%s' "$source" | jq -r "$jq_path" 2>/dev/null || echo "<jq error>")
|
|
fi
|
|
proof_assert_eq "$case_id" "$claim" "$expected" "$actual"
|
|
}
|
|
|
|
proof_skip() {
|
|
local case_id="$1" claim="$2" reason="$3"
|
|
_proof_record "$case_id" "$claim" skip "" "" "$reason"
|
|
return 0
|
|
}
|