a81291e38c
Per docs/TEST_PROOF_SCOPE.md, building the claims-verification tier
above the smoke chain. This commit lays the scaffolding and proves
the orchestrator end-to-end with one canary case (00_health).
What landed:
tests/proof/
README.md how to read a report, layout, modes
claims.yaml 24 claims enumerated (GOLAKE-001..100)
run_proof.sh orchestrator with --mode {contract|integration|performance}
and --no-bootstrap / --regenerate-{rankings,baseline}
lib/
env.sh service URLs, report dir, mode, git context
http.sh curl wrappers writing per-probe JSON + body + headers
assert.sh proof_assert_{eq,ne,contains,lt,gt,status,json_eq} +
proof_skip — each emits one JSONL record per call
metrics.sh start/stop timers, value capture, RSS sampling,
percentile compute (for Phase D)
cases/
00_health.sh canary — gateway + 6 services /health → 200,
body identifies service, latency < 500ms (21 assertions)
fixtures/
csv/workers.csv spec's 5-row deterministic CSV
text/docs.txt 4 deterministic vector docs
expected/queries.json expected results for the 5 SQL assertions
Wired into the task runner:
just proof contract # canary only this commit
just proof integration # Phase C
just proof performance # Phase D
.gitignore: /tests/proof/reports/* with !.gitkeep — same pattern as
reports/scrum/_evidence/. Per-run output is a runtime artifact.
Specs landed alongside (J's drops):
docs/TEST_PROOF_SCOPE.md the harness contract this implements
docs/CLAUDE_REFACTOR_GUARDRAILS.md process discipline this harness obeys
Verified end-to-end (cached binaries):
just proof contract wall < 2s, 21 pass / 0 fail / 0 skip
just verify wall 31s, vet + test + 9 smokes still green
Two bugs fixed during canary run, both in run_proof.sh aggregation:
- grep -c exits 1 on zero matches; the `|| echo 0` form concatenated
"0\n0" and broke jq --argjson + integer comparison. Fixed via a
_count helper that captures count-or-zero cleanly.
- per-case table iterated case scripts (filename-based) but cases
write evidence under CASE_ID. Switched to JSONL-file iteration so
multi-case scripts work and the mapping is faithful.
Phase B (contract cases) lands next: 05_embedding, 06_vector_add,
08_gateway_contracts, 09_failure_modes. Each sourcing the same lib
helpers and writing to the same report shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
52 lines
1.7 KiB
Bash
Executable File
52 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# 00_health.sh — GOLAKE-001 + GOLAKE-002.
|
|
# Verifies that gateway and each backing service answer GET /health
|
|
# with 200 and a body that includes the service name. Canonical case
|
|
# shape — copy this file when adding new cases.
|
|
|
|
set -uo pipefail
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck source=../lib/env.sh
|
|
source "${SCRIPT_DIR}/../lib/env.sh"
|
|
# shellcheck source=../lib/http.sh
|
|
source "${SCRIPT_DIR}/../lib/http.sh"
|
|
# shellcheck source=../lib/assert.sh
|
|
source "${SCRIPT_DIR}/../lib/assert.sh"
|
|
|
|
CASE_ID="GOLAKE-001-002"
|
|
CASE_NAME="health endpoints respond"
|
|
CASE_TYPE="contract"
|
|
|
|
# Allow run_proof.sh to read metadata without executing.
|
|
if [ "${1:-}" = "--metadata-only" ]; then return 0 2>/dev/null || exit 0; fi
|
|
|
|
# Each row: <name> <port>. Service name in /health body must match.
|
|
SERVICES=(
|
|
"gateway:3110"
|
|
"storaged:3211"
|
|
"catalogd:3212"
|
|
"ingestd:3213"
|
|
"queryd:3214"
|
|
"vectord:3215"
|
|
"embedd:3216"
|
|
)
|
|
|
|
for spec in "${SERVICES[@]}"; do
|
|
name="${spec%:*}"
|
|
port="${spec#*:}"
|
|
probe="${name}_health"
|
|
|
|
# Probe — captures status, body, latency to raw/http/<case>/<probe>.json
|
|
proof_get "$CASE_ID" "$probe" "http://127.0.0.1:${port}/health" >/dev/null
|
|
|
|
status=$(proof_status_of "$CASE_ID" "$probe")
|
|
body=$(proof_body_of "$CASE_ID" "$probe")
|
|
latency=$(proof_latency_of "$CASE_ID" "$probe")
|
|
|
|
proof_assert_eq "$CASE_ID" "${name} /health → 200" "200" "$status"
|
|
proof_assert_contains "$CASE_ID" "${name} body identifies service" "$name" "$body"
|
|
# Latency budget — generous so we don't get spurious failures from
|
|
# cold-start or system jitter; tighten if a real budget emerges.
|
|
proof_assert_lt "$CASE_ID" "${name} health latency < 500ms" "$latency" "500"
|
|
done
|