golangLAKEHOUSE/scripts/g1_smoke.sh
root 8b92518d21 G1P: vectord persistence to storaged + scrum (3 fixes incl. 3-way convergent)
Adds optional persistence to vectord (G1's HNSW vector search). Single-
file framed format per index — eliminates the torn-write class that
the 3-way convergent scrum finding identified:

  _vectors/<name>.lhv1  — single binary blob:
      [4 bytes magic "LHV1"]
      [4 bytes envelope_len uint32 BE]
      [envelope bytes — JSON params + metadata + version]
      [graph bytes — raw hnsw.Graph.Export]

Pre-extraction: internal/catalogd/store_client.go → internal/storeclient/
shared package, since both catalogd and vectord need it. Same pattern as
the pre-D5 catalogclient extraction.

Optional via [vectord].storaged_url config (empty = ephemeral mode).
On startup: List + Load each persisted index. After Create / batch Add /
DELETE: Save (or Delete from storaged). Save failures are logged-not-
fatal — in-memory state is the source of truth in flight.

Acceptance smoke G1P 8/8 PASS — kill+restart preserves state, post-
restart search returns dist=0 (graph round-trips exactly), DELETE
removes the file, post-delete restart shows count=0.

All 8 smokes (D1-D6 + G1 + G1P) PASS deterministically. The g1_smoke
gained scripts/g1_smoke.toml that disables persistence so the
in-memory API test stays decoupled from any rehydrate-from-storaged
state contamination.

Cross-lineage scrum on shipped code:
  - Opus 4.7 (opencode):                     1 BLOCK + 5 WARN + 3 INFO
  - Kimi K2-0905 (openrouter):               1 BLOCK + 2 WARN
  - Qwen3-coder (openrouter):                2 BLOCK + 2 WARN + 1 INFO

Fixed (3 — 1 convergent + 2 single-reviewer):
  C1 (Opus + Kimi + Qwen 3-WAY CONVERGENT WARN): Save was non-atomic
    across two PUTs — envelope-succeeds + graph-fails left a half-
    saved index that passed the "both present" List filter and
    silently mismatched metadata against vectors on Load.
    Fix: collapse to single framed file (no torn-write window
    possible).
  O-B1 (Opus BLOCK): isNotFound substring-matched "key not found"
    against the wrapped error message — brittle, any 5xx body
    containing that text would silently misclassify as missing.
    Fix: errors.Is(err, storeclient.ErrKeyNotFound).
  O-I3 (Opus INFO): handleAdd pre-validation only covered id+dim;
    NaN/Inf/zero-norm could still fail mid-batch leaving partial
    commits. Fix: extend pre-validation to call ValidateVector
    (newly exported) per item before any commit.

Dismissed (3 false positives):
  K-B1 + Q-B1 ("safeKey double-escapes %2F segments") — false
    convergent. Wire-protocol escape is decoded by storaged's chi
    router on the way in; on-disk key is the original literal.
    %2F round-trips correctly through PathEscape → URL → chi decode
    → S3 key.
  Q-B2 ("List vulnerable to race conditions") — vectord is single-
    process; no concurrent Save against List in the same vectord.

Deferred (3): rehydrate per-index timeout (G2+ multi-index scale),
saveAfter request ctx (matches G0 timeout deferral), Encode RLock
during slow writer (documented as buffer-only API).

The C1 finding is the strongest signal of the cross-lineage filter:
three independent reviewers all flagged the same torn-write hazard.
Single-file framing eliminates the class — there's now no Persistor
state where envelope and graph can disagree.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 01:33:23 -05:00

170 lines
5.9 KiB
Bash
Executable File

#!/usr/bin/env bash
# G1 smoke — vectord (HNSW vector search). All assertions go
# through gateway :3110, none direct to vectord :3215.
#
# Validates:
# - POST /v1/vectors/index → 201, params echoed back
# - POST /v1/vectors/index/{name}/add (batch of 200) → 200, length=200
# - POST /v1/vectors/index/{name}/search → top-K with the inserted
# vector at distance≈0 (recall=1 for the inserted vector itself)
# - Dim mismatch on Add → 400
# - Search on missing index → 404
# - Duplicate Create → 409
# - DELETE then Search → 404
#
# Requires vectord + gateway. Other backing services aren't needed.
#
# Usage: ./scripts/g1_smoke.sh
set -euo pipefail
cd "$(dirname "$0")/.."
export PATH="$PATH:/usr/local/go/bin"
echo "[g1-smoke] building vectord + gateway..."
go build -o bin/ ./cmd/vectord ./cmd/gateway
pkill -f "bin/vectord|bin/gateway" 2>/dev/null || true
sleep 0.3
PIDS=()
TMP="$(mktemp -d)"
cleanup() {
echo "[g1-smoke] cleanup"
for p in "${PIDS[@]}"; do [ -n "$p" ] && kill "$p" 2>/dev/null || true; done
rm -rf "$TMP"
}
trap cleanup EXIT INT TERM
poll_health() {
local port="$1" deadline=$(($(date +%s) + 5))
while [ "$(date +%s)" -lt "$deadline" ]; do
if curl -sS --max-time 1 "http://127.0.0.1:$port/health" >/dev/null 2>&1; then return 0; fi
sleep 0.05
done
return 1
}
echo "[g1-smoke] launching vectord → gateway..."
./bin/vectord -config scripts/g1_smoke.toml > /tmp/vectord.log 2>&1 &
PIDS+=($!)
poll_health 3215 || { echo "vectord failed"; tail /tmp/vectord.log; exit 1; }
./bin/gateway -config scripts/g1_smoke.toml > /tmp/gateway.log 2>&1 &
PIDS+=($!)
poll_health 3110 || { echo "gateway failed"; tail /tmp/gateway.log; exit 1; }
FAILED=0
NAME="g1_workers"
DIM=8
echo "[g1-smoke] /v1/vectors/index — create dim=$DIM:"
HTTP="$(curl -sS -o "$TMP/create.out" -w '%{http_code}' -X POST http://127.0.0.1:3110/v1/vectors/index \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$NAME\",\"dimension\":$DIM,\"distance\":\"cosine\"}")"
if [ "$HTTP" = "201" ]; then
echo " ✓ create → 201"
else
echo " ✗ create → $HTTP body=$(cat "$TMP/create.out")"; FAILED=1
fi
echo "[g1-smoke] duplicate create → 409:"
HTTP="$(curl -sS -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:3110/v1/vectors/index \
-H 'Content-Type: application/json' \
-d "{\"name\":\"$NAME\",\"dimension\":$DIM}")"
if [ "$HTTP" = "409" ]; then
echo " ✓ duplicate → 409"
else
echo " ✗ duplicate → $HTTP"; FAILED=1
fi
echo "[g1-smoke] add batch of 200 vectors:"
# Build a batch payload with 200 items. ID encodes index; vector
# is a deterministic spread so we can assert recall on a known id.
python3 - <<EOF > "$TMP/batch.json"
import json
items = []
for i in range(200):
# Each vector is unit-norm-ish with one dominant axis cycling
# through DIM dims, and a UNIQUE perturbation derived from i itself
# (no modulo — would create collisions when 200/DIM rolls over).
v = [0.0] * $DIM
v[i % $DIM] = 1.0
v[(i+1) % $DIM] = i * 0.0001 # unique per i
items.append({"id": f"w-{i:03d}", "vector": v, "metadata": {"row": i}})
print(json.dumps({"items": items}))
EOF
ADD_RESP="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/add" \
-H 'Content-Type: application/json' \
-d @"$TMP/batch.json")"
LEN="$(echo "$ADD_RESP" | jq -r '.length')"
ADDED="$(echo "$ADD_RESP" | jq -r '.added')"
if [ "$LEN" = "200" ] && [ "$ADDED" = "200" ]; then
echo " ✓ added=200, length=200"
else
echo " ✗ add → $ADD_RESP"; FAILED=1
fi
echo "[g1-smoke] search for inserted vector w-042 → recall:"
# Build the same vector w-042 was inserted with.
python3 - <<EOF > "$TMP/q.json"
import json
i = 42
v = [0.0] * $DIM
v[i % $DIM] = 1.0
v[(i+1) % $DIM] = i * 0.0001 # match the unique-per-i perturbation
print(json.dumps({"vector": v, "k": 3}))
EOF
SEARCH_RESP="$(curl -sS -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/search" \
-H 'Content-Type: application/json' \
-d @"$TMP/q.json")"
TOP_ID="$(echo "$SEARCH_RESP" | jq -r '.results[0].id')"
TOP_DIST="$(echo "$SEARCH_RESP" | jq -r '.results[0].distance')"
TOP_META="$(echo "$SEARCH_RESP" | jq -c '.results[0].metadata')"
RC="$(echo "$SEARCH_RESP" | jq -r '.results | length')"
# Distance should be ~0 (cosine of identical unit vectors). Allow 1e-4.
DIST_OK="$(python3 -c "import sys; sys.exit(0 if abs($TOP_DIST) < 1e-4 else 1)" && echo y || echo n)"
if [ "$TOP_ID" = "w-042" ] && [ "$DIST_OK" = "y" ] && [ "$RC" = "3" ] && [ "$TOP_META" = '{"row":42}' ]; then
echo " ✓ top hit = w-042 (dist=$TOP_DIST), 3 results, metadata round-tripped"
else
echo " ✗ search → top_id=$TOP_ID dist=$TOP_DIST rc=$RC meta=$TOP_META"
echo " full: $SEARCH_RESP"
FAILED=1
fi
echo "[g1-smoke] dim mismatch on add → 400:"
HTTP="$(curl -sS -o /dev/null -w '%{http_code}' -X POST "http://127.0.0.1:3110/v1/vectors/index/$NAME/add" \
-H 'Content-Type: application/json' \
-d '{"items":[{"id":"bad","vector":[1,2,3]}]}')"
if [ "$HTTP" = "400" ]; then
echo " ✓ dim mismatch → 400"
else
echo " ✗ dim mismatch → $HTTP"; FAILED=1
fi
echo "[g1-smoke] search on missing index → 404:"
HTTP="$(curl -sS -o /dev/null -w '%{http_code}' -X POST "http://127.0.0.1:3110/v1/vectors/index/no_such/search" \
-H 'Content-Type: application/json' \
-d "{\"vector\":[1,0,0,0,0,0,0,0],\"k\":1}")"
if [ "$HTTP" = "404" ]; then
echo " ✓ unknown index → 404"
else
echo " ✗ unknown index → $HTTP"; FAILED=1
fi
echo "[g1-smoke] DELETE then GET → 404:"
curl -sS -o /dev/null -X DELETE "http://127.0.0.1:3110/v1/vectors/index/$NAME"
HTTP="$(curl -sS -o /dev/null -w '%{http_code}' "http://127.0.0.1:3110/v1/vectors/index/$NAME")"
if [ "$HTTP" = "404" ]; then
echo " ✓ post-delete GET → 404"
else
echo " ✗ post-delete → $HTTP"; FAILED=1
fi
if [ "$FAILED" -eq 0 ]; then
echo "[g1-smoke] G1 acceptance gate: PASSED"
exit 0
else
echo "[g1-smoke] G1 acceptance gate: FAILED"
exit 1
fi