1ec85b0a16
Replaces single-shot baselines (40% noise floor flagged in Phase E)
with noise-aware regression detection.
What changed:
ingest n=3 runs (was 1) with 3-pass warmup
vector_add n=3 runs (was 1) with 3-pass warmup
query n=20 samples (unchanged) with 50-pass warmup
search n=20 samples (unchanged) with 50-pass warmup
RSS n=1 (unchanged — steady-state in G0)
Each metric stored as {value: median, mad: median absolute
deviation} in baseline.json (schema: v2-multisample-mad).
New regression detection:
threshold = max(3 * baseline.mad, value * 0.75)
REGRESSION iff |actual - baseline.value| > threshold AND direction
signals worse (lower throughput / higher latency).
Why these specific numbers:
3*MAD = standard "outside the spread" bound; lets high-variance
metrics tolerate their own noise.
75% floor = empirical observation: even with 50 warmups, single-
host inter-run variance on bootstrap-cold queryd was
consistently 90-130% on this box. 75% catches >75%
regressions cleanly while ignoring known noise.
lib/metrics.sh: new proof_compute_mad helper computes MAD from a
file of one-number-per-line samples. Used for both regen (to write
the baseline.mad value) and diff (read from baseline).
Honest finding from this iteration's 3 back-to-back diff runs:
query_ms shows 90-130% delta from baseline consistently — not
random noise but a systematic 2x gap between regen-time and
steady-state. The regen captured a particularly fast moment;
steady-state is slower. Operator workflow: regenerate the
baseline at a known-representative state via
`bash tests/proof/run_proof.sh --mode performance --regenerate-baseline`
rather than expecting the harness to track a moving target.
The harness's value here is the EVIDENCE RECORD (every run captures
median+MAD+p95 plus all raw samples in raw/metrics/), not the gate.
Even false-positive REGRESSION skips give operators "this run was
20ms vs baseline 10ms" which is informative.
Sample counts also written into baseline.json under "samples" so a
future audit can verify the methodology that produced the values.
Verified across 3 back-to-back runs:
ingest_rows_per_sec PASS (delta within 75%, mostly < 10%)
vectors_per_sec_add PASS
search_ms PASS
rss_* PASS
query_ms REGRESSION flagged (130/100/90%) — known
systematic gap, not bug
Closes the "40% noise floor" follow-up from Phase E FINAL_REPORT.
Honest about limitations: hard regression gating on a busy single-
host setup needs either much bigger sample counts (n≥100), longer
warmup, or moving to a dedicated benchmark host. Documented inline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
104 lines
3.5 KiB
Bash
104 lines
3.5 KiB
Bash
#!/usr/bin/env bash
|
|
# lib/metrics.sh — performance measurements for performance mode.
|
|
#
|
|
# Functions:
|
|
# proof_metric_start <case_id> <metric_name>
|
|
# proof_metric_stop <case_id> <metric_name> # writes elapsed_ms
|
|
# proof_metric_value <case_id> <metric_name> <value> [unit]
|
|
# proof_sample_rss <case_id> <process_pattern> # MB
|
|
# proof_compute_percentile <values_file> <percentile> # 50, 95, 99
|
|
#
|
|
# All metrics emit to:
|
|
# $PROOF_REPORT_DIR/raw/metrics/<case_id>.jsonl
|
|
|
|
_proof_metric_emit() {
|
|
local case_id="$1" name="$2" value="$3" unit="$4" detail="$5"
|
|
local out="${PROOF_REPORT_DIR}/raw/metrics/${case_id}.jsonl"
|
|
mkdir -p "$(dirname "$out")"
|
|
local e_detail
|
|
e_detail=$(printf '%s' "$detail" | jq -Rs .)
|
|
cat >> "$out" <<JSON
|
|
{"case_id":"${case_id}","metric":"${name}","value":${value},"unit":"${unit}","detail":${e_detail},"timestamp":"$(date -u -Iseconds)"}
|
|
JSON
|
|
}
|
|
|
|
proof_metric_start() {
|
|
local case_id="$1" name="$2"
|
|
local f="${PROOF_REPORT_DIR}/raw/metrics/_timer_${case_id}_${name}"
|
|
date +%s%3N > "$f"
|
|
}
|
|
|
|
proof_metric_stop() {
|
|
local case_id="$1" name="$2"
|
|
local f="${PROOF_REPORT_DIR}/raw/metrics/_timer_${case_id}_${name}"
|
|
if [ ! -f "$f" ]; then
|
|
echo "[metrics] timer ${name} for case ${case_id} not started" >&2
|
|
return 1
|
|
fi
|
|
local start_ms end_ms elapsed_ms
|
|
start_ms=$(cat "$f")
|
|
end_ms=$(date +%s%3N)
|
|
elapsed_ms=$((end_ms - start_ms))
|
|
rm -f "$f"
|
|
_proof_metric_emit "$case_id" "${name}_ms" "$elapsed_ms" "ms" ""
|
|
echo "$elapsed_ms"
|
|
}
|
|
|
|
proof_metric_value() {
|
|
local case_id="$1" name="$2" value="$3" unit="${4:-}"
|
|
_proof_metric_emit "$case_id" "$name" "$value" "$unit" ""
|
|
}
|
|
|
|
# Sample resident-set-size (MB) for the first matching process.
|
|
proof_sample_rss() {
|
|
local case_id="$1" pattern="$2"
|
|
local pid rss_kb rss_mb
|
|
pid=$(pgrep -f "$pattern" | head -1)
|
|
if [ -z "$pid" ]; then
|
|
_proof_metric_emit "$case_id" "rss_${pattern//\//_}_mb" 0 "MB" "process not found"
|
|
return 1
|
|
fi
|
|
rss_kb=$(awk '/^VmRSS:/ {print $2}' "/proc/$pid/status" 2>/dev/null || echo 0)
|
|
rss_mb=$(awk -v k="$rss_kb" 'BEGIN{printf "%.1f", k/1024}')
|
|
_proof_metric_emit "$case_id" "rss_${pattern//\//_}_mb" "$rss_mb" "MB" "pid=${pid}"
|
|
echo "$rss_mb"
|
|
}
|
|
|
|
# proof_compute_percentile: streams values from a file (one number per
|
|
# line), prints the requested percentile.
|
|
proof_compute_percentile() {
|
|
local file="$1" pct="$2"
|
|
sort -n "$file" | awk -v pct="$pct" '
|
|
{ v[NR] = $1 }
|
|
END {
|
|
n = NR
|
|
if (n == 0) { print "0"; exit }
|
|
idx = int((pct/100.0) * n)
|
|
if (idx < 1) idx = 1
|
|
if (idx > n) idx = n
|
|
print v[idx]
|
|
}
|
|
'
|
|
}
|
|
|
|
# proof_compute_mad: median absolute deviation. Robust noise estimator
|
|
# for skewed distributions where stddev is misleading. Output unit is
|
|
# the same as the input. Pairs naturally with the median value as
|
|
# {center, spread} for noise-aware regression detection.
|
|
#
|
|
# Definition: MAD = median(|x_i - median(x)|).
|
|
# Two passes: compute median, then median of absolute deviations.
|
|
proof_compute_mad() {
|
|
local file="$1"
|
|
if [ ! -s "$file" ]; then echo "0"; return; fi
|
|
local median
|
|
median=$(proof_compute_percentile "$file" 50)
|
|
awk -v m="$median" '{ d = ($1 > m) ? $1 - m : m - $1; print d }' "$file" \
|
|
| sort -n \
|
|
| awk '{ v[NR] = $1 } END {
|
|
n = NR; if (n == 0) { print "0"; exit }
|
|
idx = int(n / 2); if (idx < 1) idx = 1
|
|
print v[idx]
|
|
}'
|
|
}
|