golangLAKEHOUSE/reports/cutover/g5_load_test_big.md

G5 cutover slice — bigger load test (5K corpus, 200 bodies, conc-sweep + mixed)

Larger-scale follow-up to g5_load_test.md. Three axis expansions:

• Corpus: 200 → 5,000 workers
• Body variety: 6 → 200 distinct queries (4 styles × 50 fill_events rows)
• Concurrency sweep: 10 → 50 → 100 → 200, 3 minutes each
• Mixed workload: parallel embed + search at 60 conc each, 90s

All hits are direct to Go gateway on :4110/v1/matrix/search and :4110/v1/embed — no Bun frontend in this test (Bun adds proxy overhead but isn't the substrate's limit).

Setup

• Persistent Go stack on :4110+:4211-:4219 (11 daemons, 11h+ uptime at test time)
• Workers corpus: 5,000 rows from workers_500k.parquet (real production data)
• Search bodies: 200 distinct queries via gen_real_queries -limit 50 -styles all (need / client_first / looking / shorthand × 50 = 200 unique inputs to stress the embed cache + matrix retrieve path)

Concurrency sweep — /v1/matrix/search direct

Conc	Duration	Requests	RPS	p50	p95	p99	max	errors
10	3m	486,733	2,704	2.19ms	2.99ms	6.72ms	651ms	0
50	3m	1,148,543	6,381	7.08ms	14.96ms	20.20ms	77ms	0
100	3m	1,253,389	6,963	13.34ms	27.44ms	36.96ms	182ms	0
200	3m	1,460,676	8,114	23.45ms	44.20ms	56.38ms	225ms	0
Total	12 min	4,349,341	—	—	—	—	—	0

Mixed workload — /v1/embed + /v1/matrix/search in parallel

Endpoint	Conc	Duration	Requests	RPS	p50	p95	p99
/v1/embed	60	90s	1,127,854	12,531	3.31ms	10.22ms	14.59ms
/v1/matrix/search	60	90s	392,229	4,358	12.68ms	26.08ms	33.78ms
Total	120	90s	1,520,083	16,889	—	—	—

Both endpoints competing for the same matrixd / vectord / embedd processes. Zero errors across 1.52M requests in 90s.

Resource footprint during load (peak observed)

Daemon	CPU%	RSS	Read
persistent-matrixd	105%	33MB	bottleneck — pegging 1 core
persistent-gateway	44%	41MB	proxy + auth
persistent-vectord	39%	82MB	HNSW search
persistent-embedd	30%	67MB	embed cache + Ollama bridge
persistent-storaged	0.1%	22MB	idle (read-mostly)
(5 other daemons)	~0%	~25MB each	idle
Total	—	~370MB	across all 11 daemons

Compare to Rust gateway during similar load earlier today: 14.9GB RSS, 374% CPU. Go uses ~40× less memory + 4× less CPU concentration (Go spreads load across daemons; Rust packs into one mega-process).

Aggregate

Metric	Value
Total requests	5,869,424
Total wall time	~13.5 minutes
Errors (any kind)	0
Peak RSS across all 11 daemons	~370MB

5.87 million requests, zero errors. This is the substrate's production-readiness signal at scale.

What scales, what saturates

RPS scaling (search):

• 10 → 50 conc: +135% RPS, +224% p50 latency. Sub-linear scaling is expected (Little's law + Go's GMP scheduler context-switching).
• 50 → 100 conc: +9% RPS, +88% p50 latency. Saturation begins.
• 100 → 200 conc: +17% RPS, +76% p50 latency. Saturation point: matrixd is at ~105% CPU (1 core pegged); doubling concurrency past 100 adds queue depth, not throughput.

What saturates:

• matrixd is the bottleneck at conc=100+. Pegs one CPU core.
• vectord at 39% has headroom — HNSW search is fast.
• embedd at 30% has headroom — cache hit rate is high (200 bodies × millions of requests means everything stays in 4096-cap LRU).

Headroom paths (if you ever need more throughput):

1. Run multiple matrixd instances behind a load balancer. Substrate is stateless (recordings persist via storaged), so horizontal scale is straightforward.
2. Optimize matrixd's per-request work — role-gate + judge-eligibility
   • result merge. Hot path could be profile-guided.
3. Skip Bun for hot endpoints — direct nginx → Go shaved 5.7× from the original load test.

What this load test does NOT cover

• Cold-cache embed — 200 bodies × LRU cap 4096 = 100% cache hit rate after first round. Cold workloads (every query unique) would bottleneck on Ollama at ~30-50ms/embed.
• Sustained for hours — 12 minutes per endpoint. Memory/file-handle leaks would surface over multi-hour runs.
• Real coordinator demand patterns — bodies rotated round-robin; real workloads would have arrival-rate variability and burst patterns.
• Cross-daemon failure — what happens if vectord crashes mid-query? Smoke tests cover restart, but in-flight failure recovery wasn't exercised.

Repro

# Stack must be up:
./scripts/cutover/start_go_stack.sh

# Ingest 5K workers (~3 min):
./bin/staffing_workers -limit 5000 -gateway http://127.0.0.1:4110 -drop=true

# Generate 200-body search file:
go run ./scripts/cutover/gen_real_queries -limit 50 -styles all > /tmp/big_test_queries.txt
# (then convert to JSON bodies — see this doc for the python3 conversion snippet)

# Concurrency sweep:
for conc in 10 50 100 200; do
  ./bin/loadgen -url http://127.0.0.1:4110/v1/matrix/search \
    -bodies-file /tmp/big_search_bodies.txt \
    -concurrency $conc -duration 180s
done

# Mixed:
./bin/loadgen -url http://127.0.0.1:4110/v1/embed \
  -bodies-file /tmp/embed_bodies.txt \
  -concurrency 60 -duration 90s &
./bin/loadgen -url http://127.0.0.1:4110/v1/matrix/search \
  -bodies-file /tmp/big_search_bodies.txt \
  -concurrency 60 -duration 90s &
wait

Conclusion

The Go substrate handles 5.87 million requests across 13 minutes with zero errors, ~370MB total RSS, and matrixd as the visible bottleneck at concurrency-100+. Production-ready at well above any staffing-domain demand level (<1 RPS typical per coordinator, <100 RPS even with hundreds of coordinators concurrent).

The matrixd-saturates pattern is operationally good news: you know exactly which daemon to scale first if/when you grow past current demand. Substrate is well-shaped for horizontal growth.