1313eb2173
Adds the integration tier — full chain CSV→Parquet→SQL and full
text→embed→vector→search. All 10 cases (4 contract + 6 integration)
end-to-end deterministic; 8s wall total.
Cases added:
01_storage_roundtrip.sh
GOLAKE-010-012. PUT 1KiB → GET sha256-equal → LIST contains key
→ DELETE 200/204 → GET 404. Deterministic key under
proof/<case_id>/ so concurrent runs don't collide.
02_catalog_manifest.sh
GOLAKE-020-022. Fresh register existing=false → manifest read
matches → list contains dataset_id → idempotent re-register
existing=true with stable dataset_id → schema-drift register
409 (the ADR-020 contract). Per-run unique name via
PROOF_RUN_ID so existing=false is meaningful.
03_ingest_csv_to_parquet.sh
GOLAKE-030. workers.csv (5 rows) via /v1/ingest multipart →
parquet object on storaged → catalog manifest with row_count=5.
Verifies content-addressed key shape (datasets/<n>/<fp>.parquet).
04_query_correctness.sh
GOLAKE-040. The 5 SQL assertions from fixtures/expected/queries.json
against the workers fixture: count=5, Chicago=2, max=95,
safety→Barbara, Houston avg=89.5. Iterates the YAML claims, runs
each query, compares response columns to expected values.
06_vector_add_search.sh integration extension
GOLAKE-051. text → /v1/embed (4 docs from fixtures/text/docs.txt)
→ vectord add → search by query embedding. Top-1 ID per query
asserted against fixtures/expected/rankings.json. First run (or
--regenerate-rankings) writes the fixture and emits a skip with
explicit reason; subsequent runs assert against it.
07_vector_persistence_restart.sh
GOLAKE-070. add 4 unit-basis vectors → search → record top-1
distance → SIGTERM vectord → restart with the same --config →
poll /health for 8s → search again → top-1 ID and distance match
bit-identically. Skips with reason if vectord PID can't be found
or post-restart bind times out.
Two harness improvements landed alongside:
run_proof.sh writes a temp lakehouse_proof.toml with
refresh_every="500ms" override and passes --config to all booted
binaries. Production default is 30s; 04_query_correctness needs
queryd to pick up the new view within a tick. Production config
unchanged.
cleanup() now pgreps for any orphan bin/<svc> processes (anchored
to start-of-argv per memory feedback_pkill_scope.md) so a case
that restarts a service mid-run still gets cleaned up.
lib/http.sh adds proof_call(case_id, probe, method, url, args...)
— escape hatch for cases that need raw curl args (multipart -F,
custom headers). Used by 03_ingest for the multipart upload that
conflicts with proof_post's --data + Content-Type defaults.
lib/env.sh exports PROOF_RUN_ID — short unique id derived from the
report directory timestamp. Used by 02 and 07 for fresh-each-run
state isolation.
Two real findings recorded as evidence (no code changes):
- rankings.json fixture pinned: 4 queries → 4 distinct top-1 docs
via nomic-embed-text. A model swap that changes ranking now
fails the harness loudly; --regenerate-rankings is the override.
- vectord persistence kill+restart preserves top-1 distance
bit-identically — the LHV1 single-Put framed format from
G1P round-trips exactly through Save/Load.
Verified end-to-end:
just proof contract — 53 pass (4 cases)
just proof integration — 104 pass (10 cases) · 8s wall
just verify — 9 smokes still green · 33s wall
Phase D (performance baseline) lands next: 10_perf_baseline measures
rows/sec ingest, vectors/sec add, p50/p95 query+search latency, RSS,
CPU. First run writes tests/proof/baseline.json; later runs diff
against it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
69 lines
2.9 KiB
Bash
69 lines
2.9 KiB
Bash
#!/usr/bin/env bash
|
|
# lib/env.sh — proof harness environment.
|
|
#
|
|
# Sourced once by run_proof.sh and by every case script. Establishes:
|
|
# - service URLs (gateway and direct ports)
|
|
# - report directory paths
|
|
# - run context (git SHA, hostname, timestamp)
|
|
# - mode (contract|integration|performance)
|
|
#
|
|
# Cases read from these vars; never re-set them.
|
|
|
|
# Repo root — every path the harness emits is anchored here so report
|
|
# JSON is portable across reviewer machines.
|
|
PROOF_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
|
export PROOF_REPO_ROOT
|
|
|
|
# Service endpoints. Match internal/shared/config.go defaults; every
|
|
# binary binds 127.0.0.1 per the audit's R-007 mitigation.
|
|
export PROOF_GATEWAY_URL="${PROOF_GATEWAY_URL:-http://127.0.0.1:3110}"
|
|
export PROOF_STORAGED_URL="${PROOF_STORAGED_URL:-http://127.0.0.1:3211}"
|
|
export PROOF_CATALOGD_URL="${PROOF_CATALOGD_URL:-http://127.0.0.1:3212}"
|
|
export PROOF_INGESTD_URL="${PROOF_INGESTD_URL:-http://127.0.0.1:3213}"
|
|
export PROOF_QUERYD_URL="${PROOF_QUERYD_URL:-http://127.0.0.1:3214}"
|
|
export PROOF_VECTORD_URL="${PROOF_VECTORD_URL:-http://127.0.0.1:3215}"
|
|
export PROOF_EMBEDD_URL="${PROOF_EMBEDD_URL:-http://127.0.0.1:3216}"
|
|
|
|
# Mode + report directory — set by run_proof.sh before sourcing cases.
|
|
# Defaulted here so cases sourced standalone for debug still work.
|
|
export PROOF_MODE="${PROOF_MODE:-contract}"
|
|
|
|
if [ -z "${PROOF_REPORT_DIR:-}" ]; then
|
|
ts="$(date -u +%Y%m%d-%H%M%SZ)"
|
|
export PROOF_REPORT_DIR="${PROOF_REPO_ROOT}/tests/proof/reports/proof-${ts}"
|
|
fi
|
|
|
|
mkdir -p \
|
|
"${PROOF_REPORT_DIR}/raw/http" \
|
|
"${PROOF_REPORT_DIR}/raw/logs" \
|
|
"${PROOF_REPORT_DIR}/raw/outputs" \
|
|
"${PROOF_REPORT_DIR}/raw/metrics" \
|
|
"${PROOF_REPORT_DIR}/raw/cases"
|
|
|
|
# Run context — captured once per run by run_proof.sh, but recomputed
|
|
# here as fallback if a case is invoked standalone.
|
|
if [ ! -f "${PROOF_REPORT_DIR}/raw/context.json" ]; then
|
|
git_sha="$(cd "$PROOF_REPO_ROOT" && git rev-parse HEAD 2>/dev/null || echo unknown)"
|
|
git_dirty="$(cd "$PROOF_REPO_ROOT" && [ -n "$(git status --porcelain 2>/dev/null)" ] && echo true || echo false)"
|
|
cat > "${PROOF_REPORT_DIR}/raw/context.json" <<JSON
|
|
{
|
|
"git_sha": "${git_sha}",
|
|
"git_dirty": ${git_dirty},
|
|
"hostname": "$(hostname)",
|
|
"timestamp_utc": "$(date -u -Iseconds)",
|
|
"mode": "${PROOF_MODE}",
|
|
"harness_version": "1"
|
|
}
|
|
JSON
|
|
fi
|
|
|
|
export PROOF_GIT_SHA="$(cd "$PROOF_REPO_ROOT" && git rev-parse HEAD 2>/dev/null || echo unknown)"
|
|
|
|
# A short unique id per orchestrator run, used by cases that need
|
|
# fresh-each-run state (e.g. catalog dataset names that should be
|
|
# absent on first register). Derived from the report dir basename so
|
|
# all cases in one run share the same ID. Strip the "proof-" prefix
|
|
# and dashes; use last 8 chars for brevity.
|
|
_run_basename="$(basename "$PROOF_REPORT_DIR" | sed 's/proof-//; s/-//g; s/Z$//')"
|
|
export PROOF_RUN_ID="${_run_basename: -8}"
|