a81291e38c
Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
83 lines
2.7 KiB
Bash
83 lines
2.7 KiB
Bash
#!/usr/bin/env bash
|
|
# lib/metrics.sh — performance measurements for performance mode.
|
|
#
|
|
# Functions:
|
|
# proof_metric_start <case_id> <metric_name>
|
|
# proof_metric_stop <case_id> <metric_name> # writes elapsed_ms
|
|
# proof_metric_value <case_id> <metric_name> <value> [unit]
|
|
# proof_sample_rss <case_id> <process_pattern> # MB
|
|
# proof_compute_percentile <values_file> <percentile> # 50, 95, 99
|
|
#
|
|
# All metrics emit to:
|
|
# $PROOF_REPORT_DIR/raw/metrics/<case_id>.jsonl
|
|
|
|
_proof_metric_emit() {
|
|
local case_id="$1" name="$2" value="$3" unit="$4" detail="$5"
|
|
local out="${PROOF_REPORT_DIR}/raw/metrics/${case_id}.jsonl"
|
|
mkdir -p "$(dirname "$out")"
|
|
local e_detail
|
|
e_detail=$(printf '%s' "$detail" | jq -Rs .)
|
|
cat >> "$out" <<JSON
|
|
{"case_id":"${case_id}","metric":"${name}","value":${value},"unit":"${unit}","detail":${e_detail},"timestamp":"$(date -u -Iseconds)"}
|
|
JSON
|
|
}
|
|
|
|
proof_metric_start() {
|
|
local case_id="$1" name="$2"
|
|
local f="${PROOF_REPORT_DIR}/raw/metrics/_timer_${case_id}_${name}"
|
|
date +%s%3N > "$f"
|
|
}
|
|
|
|
proof_metric_stop() {
|
|
local case_id="$1" name="$2"
|
|
local f="${PROOF_REPORT_DIR}/raw/metrics/_timer_${case_id}_${name}"
|
|
if [ ! -f "$f" ]; then
|
|
echo "[metrics] timer ${name} for case ${case_id} not started" >&2
|
|
return 1
|
|
fi
|
|
local start_ms end_ms elapsed_ms
|
|
start_ms=$(cat "$f")
|
|
end_ms=$(date +%s%3N)
|
|
elapsed_ms=$((end_ms - start_ms))
|
|
rm -f "$f"
|
|
_proof_metric_emit "$case_id" "${name}_ms" "$elapsed_ms" "ms" ""
|
|
echo "$elapsed_ms"
|
|
}
|
|
|
|
proof_metric_value() {
|
|
local case_id="$1" name="$2" value="$3" unit="${4:-}"
|
|
_proof_metric_emit "$case_id" "$name" "$value" "$unit" ""
|
|
}
|
|
|
|
# Sample resident-set-size (MB) for the first matching process.
|
|
proof_sample_rss() {
|
|
local case_id="$1" pattern="$2"
|
|
local pid rss_kb rss_mb
|
|
pid=$(pgrep -f "$pattern" | head -1)
|
|
if [ -z "$pid" ]; then
|
|
_proof_metric_emit "$case_id" "rss_${pattern//\//_}_mb" 0 "MB" "process not found"
|
|
return 1
|
|
fi
|
|
rss_kb=$(awk '/^VmRSS:/ {print $2}' "/proc/$pid/status" 2>/dev/null || echo 0)
|
|
rss_mb=$(awk -v k="$rss_kb" 'BEGIN{printf "%.1f", k/1024}')
|
|
_proof_metric_emit "$case_id" "rss_${pattern//\//_}_mb" "$rss_mb" "MB" "pid=${pid}"
|
|
echo "$rss_mb"
|
|
}
|
|
|
|
# proof_compute_percentile: streams values from a file (one number per
|
|
# line), prints the requested percentile.
|
|
proof_compute_percentile() {
|
|
local file="$1" pct="$2"
|
|
sort -n "$file" | awk -v pct="$pct" '
|
|
{ v[NR] = $1 }
|
|
END {
|
|
n = NR
|
|
if (n == 0) { print "0"; exit }
|
|
idx = int((pct/100.0) * n)
|
|
if (idx < 1) idx = 1
|
|
if (idx > n) idx = n
|
|
print v[idx]
|
|
}
|
|
'
|
|
}
|