b314ed1c94
Today's sidecar drop (lakehouse ba928b1) changed Rust's embed
transport from gateway → sidecar → Ollama (2 hops) to gateway →
Ollama directly. Go's embedd has always been direct. A drift here
would mean: same query, different vector → different HNSW top-K →
different staffing recommendations. This probe is the regression
gate for that surface.
Fixtures cover staffing-domain shapes (forklift, welder, OSHA,
dental, CNC) plus stress shapes (unicode "Café résumé ⭐ 你好",
single char "x", 200-word long fixture).
Match metric: cosine similarity ≥ 0.99999. Byte-equal isn't
expected — Go round-trips through []float32 internally while Rust
stays at Vec<f64>, so JSON serialization introduces small float
drift. What matters operationally is vector direction (HNSW uses
cosine distance), and both runtimes preserve it when calling the
same Ollama with the same model.
Result: **8/8 fixtures match** including the long + unicode cases.
Sidecar drop didn't disturb the embed surface. The probe also
forces both endpoints to use `nomic-embed-text` so the v1-vs-v2-moe
default difference doesn't pollute the comparison.
5th cross-runtime parity probe joining the family:
- validator_parity (6/6)
- extract_json_parity (12/12)
- session_log_parity (4/4)
- materializer_parity (2/2)
- embed_parity (8/8) — this commit
Cumulative: 32/32 parity assertions across 5 probes covering
HTTP shape (validator, embed), CLI output (materializer), unit
behavior (extract_json), and persisted shape (session_log).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
138 lines
4.7 KiB
Bash
Executable File
138 lines
4.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# embed_parity — verify Rust /ai/embed (now direct Ollama post-sidecar
|
|
# drop) and Go /v1/embed (via embedd) produce equivalent 768-dim
|
|
# vectors for identical input.
|
|
#
|
|
# Why: today's sidecar drop (lakehouse commit ba928b1) changed Rust's
|
|
# embed transport. Go's embedd has always been direct. A drift would
|
|
# break retrieval semantics — same query, different vector → different
|
|
# top-K matches → different staffing recommendations. This probe is
|
|
# the regression gate.
|
|
#
|
|
# Match metric: cosine similarity > 0.99999. Byte-equal comparison
|
|
# isn't realistic — Go represents embeddings as []float32 and Rust
|
|
# as Vec<f64>, so JSON round-trip introduces small float drift. What
|
|
# matters for retrieval correctness is cosine direction, which both
|
|
# runtimes preserve when calling the same Ollama with the same model.
|
|
#
|
|
# Forces both endpoints to use `nomic-embed-text` so the v1-vs-v2-moe
|
|
# default difference doesn't pollute the comparison.
|
|
#
|
|
# Outputs: reports/cutover/gauntlet_2026-05-02/parity/embed_parity.md
|
|
|
|
set -uo pipefail
|
|
cd "$(dirname "$0")/../../.."
|
|
|
|
RUST_GW="${RUST_GW:-http://127.0.0.1:3100}"
|
|
GO_GW="${GO_GW:-http://127.0.0.1:4110}"
|
|
MODEL="${EMBED_MODEL:-nomic-embed-text}"
|
|
THRESHOLD="${EMBED_COSINE_THRESHOLD:-0.99999}"
|
|
OUT_DIR="reports/cutover/gauntlet_2026-05-02/parity"
|
|
mkdir -p "$OUT_DIR"
|
|
OUT="$OUT_DIR/embed_parity.md"
|
|
|
|
# Fixtures cover the staffing-domain text shapes plus stress shapes
|
|
# (unicode, very short, very long).
|
|
FIXTURES=(
|
|
"forklift operator"
|
|
"Welder, Toledo OH, 2nd shift"
|
|
"OSHA-30 certified driver"
|
|
"dental hygienist with 3 years experience"
|
|
"CNC machine operator graveyard"
|
|
"Café résumé ⭐ 你好"
|
|
"x"
|
|
"$(printf 'long fixture: %.0s' {1..200})end"
|
|
)
|
|
|
|
probe() {
|
|
local gw="$1" path="$2" text="$3"
|
|
curl -sf -m 15 -X POST "$gw$path" \
|
|
-H 'Content-Type: application/json' \
|
|
-d "$(jq -nc --arg t "$text" --arg m "$MODEL" '{texts:[$t], model:$m}')" 2>/dev/null
|
|
}
|
|
|
|
# Cosine via inline python3 (bash arithmetic doesn't have sqrt).
|
|
cosine() {
|
|
local rust_json="$1" go_json="$2"
|
|
python3 - <<EOF
|
|
import json, math, sys
|
|
r = json.loads('''$rust_json''')
|
|
g = json.loads('''$go_json''')
|
|
# Rust /ai/embed returns {embeddings: [[...]]}, dimensions field
|
|
# Go /v1/embed returns {vectors: [[...]]}, dimension field
|
|
rv = r.get("embeddings", r.get("vectors", []))[0]
|
|
gv = g.get("vectors", g.get("embeddings", []))[0]
|
|
if len(rv) != len(gv):
|
|
print(f"DIM_MISMATCH:rust={len(rv)},go={len(gv)}")
|
|
sys.exit(0)
|
|
dot = sum(a*b for a,b in zip(rv,gv))
|
|
nr = math.sqrt(sum(a*a for a in rv))
|
|
ng = math.sqrt(sum(b*b for b in gv))
|
|
if nr == 0 or ng == 0:
|
|
print("ZERO_NORM")
|
|
sys.exit(0)
|
|
print(f"{dot/(nr*ng):.10f}")
|
|
EOF
|
|
}
|
|
|
|
TOTAL=0; MATCH=0; DIFF=0
|
|
DIFF_DETAIL=""
|
|
|
|
for text in "${FIXTURES[@]}"; do
|
|
TOTAL=$((TOTAL+1))
|
|
rust_json=$(probe "$RUST_GW" "/ai/embed" "$text")
|
|
go_json=$(probe "$GO_GW" "/v1/embed" "$text")
|
|
if [ -z "$rust_json" ] || [ -z "$go_json" ]; then
|
|
DIFF=$((DIFF+1))
|
|
label=$(echo "$text" | head -c 60)
|
|
DIFF_DETAIL="$DIFF_DETAIL"$'\n'"- \`$label\`: helper failed (rust_empty=$([[ -z "$rust_json" ]] && echo y || echo n) go_empty=$([[ -z "$go_json" ]] && echo y || echo n))"
|
|
continue
|
|
fi
|
|
cos=$(cosine "$rust_json" "$go_json")
|
|
label=$(echo "$text" | head -c 60 | tr '\n' ' ')
|
|
case "$cos" in
|
|
DIM_MISMATCH:*|ZERO_NORM)
|
|
DIFF=$((DIFF+1))
|
|
DIFF_DETAIL="$DIFF_DETAIL"$'\n'"- \`$label\` → $cos"
|
|
;;
|
|
*)
|
|
# Compare cos vs threshold via awk (bash can't compare floats).
|
|
if awk -v c="$cos" -v t="$THRESHOLD" 'BEGIN{exit !(c >= t)}'; then
|
|
MATCH=$((MATCH+1))
|
|
else
|
|
DIFF=$((DIFF+1))
|
|
DIFF_DETAIL="$DIFF_DETAIL"$'\n'"- \`$label\` → cos=$cos (below threshold $THRESHOLD)"
|
|
fi
|
|
;;
|
|
esac
|
|
done
|
|
|
|
{
|
|
echo "# /v1/embed cross-runtime parity probe"
|
|
echo
|
|
echo "**Date:** $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
echo "**Rust:** \`$RUST_GW/ai/embed\` · **Go:** \`$GO_GW/v1/embed\`"
|
|
echo "**Model:** \`$MODEL\` (forced — overrides each side's default)"
|
|
echo "**Match metric:** cosine similarity ≥ \`$THRESHOLD\`"
|
|
echo
|
|
echo "Identical text → both endpoints. Cosine compares vector"
|
|
echo "DIRECTION (the operationally-meaningful property for HNSW"
|
|
echo "retrieval); byte-equal isn't expected because Go round-trips"
|
|
echo "through float32 internally while Rust stays at f64."
|
|
echo
|
|
echo "**Tally:** $MATCH match · $DIFF diff (out of $TOTAL fixtures)"
|
|
if [ -n "$DIFF_DETAIL" ]; then
|
|
echo
|
|
echo "## Divergences"
|
|
echo "$DIFF_DETAIL"
|
|
else
|
|
echo
|
|
echo "_Cosine ≥ $THRESHOLD on every fixture — embed parity holds_"
|
|
echo "_post-sidecar-drop. Rust and Go produce vectors that point in_"
|
|
echo "_the same direction in 768-dim space._"
|
|
fi
|
|
} > "$OUT"
|
|
|
|
echo "[parity] embed: $MATCH match / $DIFF diff (out of $TOTAL) → $OUT"
|
|
[ "$DIFF" -eq 0 ]
|