root 6d18394416 proof harness Phase B: 4 contract cases · 53/0/1 green
Added the contract tier above 00_health canary. All 5 contract cases
now cover GOLAKE-001-003, 050, 060-061, 080-085 — 53 assertions pass,
1 informational skip, 0 fail. Wall: 4s end-to-end (cached binaries).

Cases:
  05_embedding_contract.sh
    GOLAKE-050. POST /v1/embed with one short text → asserts dim=768,
    one vector returned, vector length matches dimension, sum of
    squared elements > 0 (proxy for non-zero), response.model echoed.
    Skips with explicit reason if Ollama is unreachable (502 from
    embedd) — per spec hard rule "skipped tests do not appear as
    passed."

  06_vector_add_search.sh
    GOLAKE-060 + GOLAKE-061. Synthetic dim=4 unit basis vectors.
    Create index → add 3 vectors → get-index returns length=3 →
    search([1,0,0,0],k=3) returns v1 at rank 1 with distance < 0.001.
    Cleanup with DELETE. No embedd dependency — pure contract layer.

  08_gateway_contracts.sh
    GOLAKE-003. For each /v1/* route, asserts gateway and direct
    upstream return identical status AND identical response body
    (sha256 match). Confirms gateway is a proxy not a transformer.
    Status passthrough verified on both 200 path (storage/list,
    catalog/list) and 4xx path (sql empty body → 400 from queryd).

  09_failure_modes.sh
    GOLAKE-080..085. Six failure-mode contracts:
      080 malformed JSON → 4xx on catalog/ingest/sql/embed
      081 missing required field → 4xx on catalog/vectors/embed
      082 bad SQL → 4xx with non-empty error body
      083 vector dim mismatch → 4xx
      084 missing storage object → 404
      085 duplicate vector ID → INFORMATIONAL (spec says required:false)
          first/second statuses recorded as evidence; contract decided
          later from the recorded record.

Two new lib helpers in lib/assert.sh:
  proof_assert_status_in <id> <claim> "200 201 204" <probe>
    pass if status is in the space-separated list. Used for
    delete-returns-200-or-204 case where vectord returns 204.

  proof_assert_status_4xx <id> <claim> <probe>
    pass if status in [400, 500). Used for failure modes where the
    specific 4xx code may vary (400 vs 422 vs 409). Records actual
    code as evidence.

Two real contract findings recorded by the harness during build:
  - vectord add expects {"items": [...]}, not {"vectors": [...]}.
    My initial test sent the wrong field; would have masked the bug
    forever in CI. The harness caught it via the assertion failure.
  - vectord create returns 201 Created, delete returns 204 No Content.
    Documented in the test fixtures as canonical.

Regression: just verify wall 33s, vet + test + 9 smokes still green.

Phase C (integration) lands next: 01_storage_roundtrip, 02_catalog_manifest,
03_ingest_csv_to_parquet, 04_query_correctness, 05/06 integration extends,
07_vector_persistence_restart.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 05:15:04 -05:00

154 lines
5.9 KiB
Bash

#!/usr/bin/env bash
# lib/assert.sh — assertions that record evidence per the spec.
#
# Each assertion appends one record to:
# $PROOF_REPORT_DIR/raw/cases/<case_id>.jsonl
#
# Each line is a self-describing JSON object — case ID, claim, expected,
# actual, result {pass|fail|skip}, optional evidence pointers. Cases
# call multiple assertions; run_proof.sh aggregates JSONL → summary.
#
# Functions:
# proof_assert_eq <case_id> <claim> <expected> <actual>
# proof_assert_ne <case_id> <claim> <not_expected> <actual>
# proof_assert_contains <case_id> <claim> <substring> <haystack>
# proof_assert_lt <case_id> <claim> <a> <b> # passes if a < b
# proof_assert_gt <case_id> <claim> <a> <b> # passes if a > b
# proof_assert_status <case_id> <claim> <expected_status> <probe_name>
# proof_assert_json_eq <case_id> <claim> <jq_path> <expected> <body_or_file>
# proof_skip <case_id> <claim> <reason>
#
# All return 0 (case scripts decide their own halt-on-fail policy).
_proof_record() {
local case_id="$1" claim="$2" result="$3" expected="$4" actual="$5" detail="$6"
local out="${PROOF_REPORT_DIR}/raw/cases/${case_id}.jsonl"
mkdir -p "$(dirname "$out")"
# JSON-escape the variable inputs.
local e_claim e_expected e_actual e_detail
e_claim=$(printf '%s' "$claim" | jq -Rs .)
e_expected=$(printf '%s' "$expected" | jq -Rs .)
e_actual=$(printf '%s' "$actual" | jq -Rs .)
e_detail=$(printf '%s' "$detail" | jq -Rs .)
local ts
ts=$(date -u -Iseconds)
cat >> "$out" <<JSON
{"case_id":"${case_id}","claim":${e_claim},"result":"${result}","expected":${e_expected},"actual":${e_actual},"detail":${e_detail},"timestamp":"${ts}","git_sha":"${PROOF_GIT_SHA}"}
JSON
}
proof_assert_eq() {
local case_id="$1" claim="$2" expected="$3" actual="$4"
if [ "$expected" = "$actual" ]; then
_proof_record "$case_id" "$claim" pass "$expected" "$actual" ""
return 0
fi
_proof_record "$case_id" "$claim" fail "$expected" "$actual" "values differ"
return 0
}
proof_assert_ne() {
local case_id="$1" claim="$2" not_expected="$3" actual="$4"
if [ "$not_expected" != "$actual" ]; then
_proof_record "$case_id" "$claim" pass "!= ${not_expected}" "$actual" ""
return 0
fi
_proof_record "$case_id" "$claim" fail "!= ${not_expected}" "$actual" "values matched (should differ)"
return 0
}
proof_assert_contains() {
local case_id="$1" claim="$2" substring="$3" haystack="$4"
if [[ "$haystack" == *"$substring"* ]]; then
_proof_record "$case_id" "$claim" pass "contains: ${substring}" "$haystack" ""
return 0
fi
_proof_record "$case_id" "$claim" fail "contains: ${substring}" "$haystack" "substring not found"
return 0
}
proof_assert_lt() {
local case_id="$1" claim="$2" a="$3" b="$4"
# awk handles ints + floats uniformly
if awk -v a="$a" -v b="$b" 'BEGIN{exit !(a < b)}'; then
_proof_record "$case_id" "$claim" pass "${a} < ${b}" "${a}" ""
return 0
fi
_proof_record "$case_id" "$claim" fail "${a} < ${b}" "${a}" "${a} is not less than ${b}"
return 0
}
proof_assert_gt() {
local case_id="$1" claim="$2" a="$3" b="$4"
if awk -v a="$a" -v b="$b" 'BEGIN{exit !(a > b)}'; then
_proof_record "$case_id" "$claim" pass "${a} > ${b}" "${a}" ""
return 0
fi
_proof_record "$case_id" "$claim" fail "${a} > ${b}" "${a}" "${a} is not greater than ${b}"
return 0
}
# proof_assert_status compares the status from a previously-recorded
# probe against an expected value. Probe must have run via lib/http.sh.
proof_assert_status() {
local case_id="$1" claim="$2" expected="$3" probe_name="$4"
local actual
actual=$(proof_status_of "$case_id" "$probe_name" 2>/dev/null || echo missing)
proof_assert_eq "$case_id" "$claim" "$expected" "$actual"
}
# proof_assert_json_eq: jq-based equality on response body or a file.
# body_or_file: if starts with @, read from file; otherwise treat as
# literal JSON string.
proof_assert_json_eq() {
local case_id="$1" claim="$2" jq_path="$3" expected="$4" source="$5"
local actual
if [[ "$source" == @* ]]; then
actual=$(jq -r "$jq_path" "${source#@}" 2>/dev/null || echo "<jq error>")
else
actual=$(printf '%s' "$source" | jq -r "$jq_path" 2>/dev/null || echo "<jq error>")
fi
proof_assert_eq "$case_id" "$claim" "$expected" "$actual"
}
proof_skip() {
local case_id="$1" claim="$2" reason="$3"
_proof_record "$case_id" "$claim" skip "" "" "$reason"
return 0
}
# proof_assert_status_in: pass if probe's status is in the space-separated
# expected list. Use when a route legitimately has multiple OK codes (e.g.
# 200 vs 204 vs 201 across services). Records a clean pass/fail with the
# actual status echoed back.
proof_assert_status_in() {
local case_id="$1" claim="$2" expected_list="$3" probe="$4"
local actual found
actual=$(proof_status_of "$case_id" "$probe" 2>/dev/null || echo missing)
found=0
for ok in $expected_list; do
[ "$ok" = "$actual" ] && { found=1; break; }
done
if [ "$found" = 1 ]; then
_proof_record "$case_id" "$claim" pass "in {${expected_list}}" "$actual" ""
else
_proof_record "$case_id" "$claim" fail "in {${expected_list}}" "$actual" "status not in expected list"
fi
return 0
}
# proof_assert_status_4xx: pass if probe's status is in [400, 500). Use
# for failure-mode contracts where the specific 4xx code is allowed to
# vary (400 vs 422 vs 409) — only "is a client error" matters.
proof_assert_status_4xx() {
local case_id="$1" claim="$2" probe="$3"
local actual
actual=$(proof_status_of "$case_id" "$probe" 2>/dev/null || echo missing)
if awk -v s="$actual" 'BEGIN{exit !(s+0 >= 400 && s+0 < 500)}'; then
_proof_record "$case_id" "$claim" pass "4xx" "$actual" ""
else
_proof_record "$case_id" "$claim" fail "4xx" "$actual" "status is not in 400-499"
fi
return 0
}