profit/golangLAKEHOUSE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
golangLAKEHOUSE/reports/cutover/g5_load_test.md
root c164a3da96 g5 cutover: production load test — 0 errors / 101k req · Go direct = 2,772 RPS 
Sustained-traffic load test against the cutover slice. Three runs,
zero correctness errors across 101,770 total requests. Substrate
holds up under concurrent load — matrix gate, vectord HNSW,
embedd cache, gateway proxy all hold. This was the load test's
primary question; latency numbers are secondary.

scripts/cutover/loadgen — focused Go load generator. 6-query
rotating body mix (Forklift/CNC/Warehouse/Picker/Loader/Shipping).
Configurable URL/concurrency/duration. Reports per-status-code
counts + p50/p95/p99 latencies + JSON summary on stderr.

Three runs:

  baseline (Bun → Go, conc=1, 10s):
    4,085 req · 408 RPS · p50 1.3ms · p99 32ms · max 215ms

  sustained (Bun → Go, conc=10, 30s):
    14,527 req · 484 RPS · p50 4.6ms · p99 92ms · max 372ms

  direct (→ Go, conc=10, 30s):
    83,158 req · 2,772 RPS · p50 2.5ms · p99 8.5ms · max 16ms

Critical findings:

1. ZERO correctness errors across 101k requests. No 5xx, no
   transport errors, no panics. Concurrency-safety verified across
   matrix gate / vectord / gateway / embedd cache.

2. Direct-to-Go is production-grade. 2,772 RPS at p99 8.5ms on a
   single host, no scaling cliff at concurrency=10.

3. Bun frontend is the bottleneck. -82% RPS, +982% p99 vs direct.
   Single-process JS event loop queueing under concurrent
   requests — known Bun proxy-mode characteristic. The substrate
   itself isn't the limiter.

4. For staffing-domain demand levels (<1 RPS typical per
   coordinator), Bun-fronted 484 RPS has 480× headroom. No
   urgency to optimize Bun out of the data path. If/when
   concurrent demand grows orders of magnitude, the path is
   nginx → Go direct for hot endpoints, skip Bun.

Substrate is now load-tested and verified production-ready.

What this load test does NOT cover (documented in
g5_load_test.md): cold-cache embed, larger corpus, mixed
read/write, multi-host, full 5-loop traffic with judge gate
calls. Each is its own probe shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 04:20:41 -05:00

4.6 KiB
Raw Blame History

G5 cutover slice — production load test

Sustained-traffic load test against the cutover slice. Companion to g5_first_loop_live.md (which proved learning-loop math) — this report proves the substrate holds up under concurrent load.

Setup

  • Persistent Go stack on :4110+:4211-:4219 (11 daemons)
  • Workers corpus: 200 rows, in-memory + persisted to MinIO
  • Bun mcp-server on :3700 with GO_LAKEHOUSE_URL=http://127.0.0.1:4110
  • Load generator: scripts/cutover/loadgen/ — Go binary, 6-query rotating body mix (Forklift/CNC/Warehouse/Picker/Loader/Shipping)
  • All queries use_playbook=false (cold-pass retrieval only — the load test isolates retrieval performance from learning-loop costs)

Results

Run Path Concurrency Duration Requests RPS p50 p95 p99 max errors
1 Bun /_go/* → Go 1 10s 4,085 408 1.3ms 3.2ms 32ms 215ms 0
2 Bun /_go/* → Go 10 30s 14,527 484 4.6ms 76ms 92ms 372ms 0
3 Direct → Go (:4110) 10 30s 83,158 2,772 2.5ms 7.2ms 8.5ms 16ms 0

Total: 101,770 requests, zero errors.

Read

What the load test confirmed

  1. Zero correctness errors across 101k requests. Matrix gate + vectord HNSW + embedd cache + gateway proxy all hold under sustained concurrent traffic. No 5xx, no transport errors, no panics. This was the load test's primary question.

  2. Direct-to-Go performance is production-grade. 2,772 RPS at p50 2.5ms / p99 8.5ms / max 16ms on a single host. The substrate itself has no scaling cliff at concurrency=10.

  3. The substrate's tail latency is well-bounded direct. p99 8.5ms means 99% of requests complete in under 9ms. For a vector- search workload (which involves embed → HNSW search → metadata join), that's a strong number.

What the load test exposed

Bun frontend is the bottleneck. Adding Bun's reframing layer collapses throughput by 5.7× and inflates p99 by 11×:

Metric Direct Via Bun Cost
RPS 2,772 484 -82%
p50 latency 2.5ms 4.6ms +84%
p99 latency 8.5ms 92ms +982%
max latency 16ms 372ms +2,225%

The p99/max cliff (>10× worse via Bun) suggests Bun's single-process JS event loop is queueing under concurrent requests. This is a known characteristic of Node/Bun in proxy mode — the event loop serializes I/O completions, and at concurrency=10 the queue depth during fan-out shows up as tail-latency cliffs.

What this means for production

For staffing-domain demand levels (single-coordinator workflows typically run <1 RPS even at peak), the Bun-fronted 484 RPS path has 480× headroom. No urgency to optimize Bun out of the data path.

If/when concurrent demand grows orders of magnitude (e.g. 100+ simultaneous coordinators, automated pipelines), the optimization path is clear: route nginx → Go directly for /v1/matrix/search (or other hot endpoints), skip Bun for those. The 5.7× throughput gain isn't gated on Go-side optimization — it's gated on Bun reframing exit.

The substrate itself is production-ready. Zero errors, sub-10ms p99 direct, no concurrency bugs surfaced under sustained load. The load test's null result on correctness is the load test's signal.

What this load test does NOT cover

  • Embedder hot path: bodies rotate across 6 queries, so embed cache hits frequently. Cold-cache RPS would be lower.
  • Larger corpus: 200 workers is a small index. HNSW search costs scale with O(log n) so 5K or 500K row corpora would show small additional latency, but the experiment isn't done.
  • Mixed read/write: load is read-only. Concurrent ingest+search hasn't been tested under sustained load.
  • Multi-host cluster: single-process load on one box. Horizontal scaling characteristics unknown.
  • Real chatd/observer/pathway calls: load test bodies set use_playbook=false to isolate the matrix→vectord retrieve path. Full 5-loop traffic (with playbook lookup + judge gate) has different RPS characteristics.

Repro

# Stack must be up:
./scripts/cutover/start_go_stack.sh
./bin/staffing_workers -limit 200 -gateway http://127.0.0.1:4110 -drop=true

# Build loadgen:
go build -o bin/loadgen ./scripts/cutover/loadgen

# Three runs:
./bin/loadgen -url http://localhost:3700/_go/v1/matrix/search -concurrency 1  -duration 10s
./bin/loadgen -url http://localhost:3700/_go/v1/matrix/search -concurrency 10 -duration 30s
./bin/loadgen -url http://localhost:4110/v1/matrix/search       -concurrency 10 -duration 30s

JSON summary on stderr is parseable for CI integration.