e8cf113af8
Production-readiness gauntlet exploiting the dual Rust/Go
implementation as a measurement instrument.
## Phase 1 — Full smoke chain
21/21 PASS in ~60s. Substrate intact across the full service surface.
## Phase 2 — Per-component scrum (token-volume fix)
Prior wave (165KB diff): Kimi 62 tokens out, Qwen 297 → no useful
analysis. This wave splits today's commits into 4 focused bundles
(36-71KB each):
c1 validatord (46KB) → 0 convergent / 11 distinct
c2 vectord substrate (36KB) → 0 convergent / 10 distinct
c3 materializer (71KB) → 0 convergent / 6 distinct (Opus emitted
a BLOCK then self-retracted in same response)
c4 replay (45KB) → 0 convergent / 10 distinct
Reviewer engagement vs prior wave: Kimi went 62 → ~250 tokens out
once bundles dropped below 60KB.
scripts/scrum_review.sh hardening:
* Diff-size guard (warn >60KB, hard-fail >100KB,
SCRUM_FORCE_OVERSIZE=1 override)
* Tightened prompt — file path must appear EXACTLY as in diff
so post-processor can grep WHERE: lines reliably
* Auto-tally step dedupes by (reviewer, location); convergence
counts distinct lineages (closes the prior `opus+opus+opus`
false-convergence bug)
## Phase 3 — Cross-runtime validator parity probe (the headline finding)
scripts/cutover/parity/validator_parity.sh sends 6 identical
/v1/validate cases to Rust :3100 AND Go :4110, compares status+body.
Result: **6/6 status codes match · 5/6 body shapes diverge.**
Rust returns serde-tagged enum: {"Schema":{"field":"x","reason":"y"}}
Go returns flat exported-fields: {"Kind":"schema","Field":"x","Reason":"y"}
Both round-trip inside their own runtime; a caller swapping one for
the other would break parsing silently. Captured as new _open_ row
in docs/ARCHITECTURE_COMPARISON.md decisions tracker.
This is the "use the dual-implementation as a measurement instrument"
return — single-repo scrums can't catch this class of cross-runtime
drift.
## Phase 4 — Production assessment
ship-with-known-gap. Validator wire-format gap is documented, not
regressed. ~50 LOC future fix on Go side (custom MarshalJSON on
ValidationError to match Rust's serde shape).
Persistent stack config (/tmp/lakehouse-persistent.toml) gains
validatord on :3221 + persistent-validatord binary so operators
bringing up the persistent stack get the new daemon automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
333 lines
16 KiB
Plaintext
333 lines
16 KiB
Plaintext
[d1-smoke] building...
|
|
[d1-smoke] launching in dep order...
|
|
[d1-smoke] /health probes:
|
|
✓ gateway (:3110) → {"status":"ok","service":"gateway"}
|
|
✓ storaged (:3211) → {"status":"ok","service":"storaged"}
|
|
✓ catalogd (:3212) → {"status":"ok","service":"catalogd"}
|
|
✓ ingestd (:3213) → {"status":"ok","service":"ingestd"}
|
|
✓ queryd (:3214) → {"status":"ok","service":"queryd"}
|
|
[d1-smoke] gateway proxy probes (D6+):
|
|
✓ POST /v1/ingest (no name) → 400 from ingestd (proxy wired)
|
|
✓ POST /v1/sql (no body) → 400 from queryd (proxy wired)
|
|
[d1-smoke] D1 acceptance gate: PASSED
|
|
[d1-smoke] cleanup
|
|
[d2-smoke] building storaged...
|
|
[d2-smoke] launching storaged...
|
|
[d2-smoke] PUT round-trip:
|
|
✓ PUT d2-smoke/1777712027.bin → 200
|
|
[d2-smoke] GET echoes bytes:
|
|
✓ GET d2-smoke/1777712027.bin → bytes match
|
|
[d2-smoke] LIST includes key:
|
|
✓ LIST prefix=d2-smoke/ → contains d2-smoke/1777712027.bin
|
|
[d2-smoke] DELETE then GET → 404:
|
|
✓ DELETE then GET → 404
|
|
[d2-smoke] 256 MiB cap → 413:
|
|
✓ PUT 257 MiB → 413
|
|
[d2-smoke] semaphore: 5th concurrent PUT → 503 + Retry-After:5
|
|
✓ 5th concurrent PUT → 503 + Retry-After: 5
|
|
[d2-smoke] D2 acceptance gate: PASSED
|
|
[d2-smoke] cleanup
|
|
[d3-smoke] building storaged + catalogd...
|
|
[d3-smoke] launching storaged...
|
|
[d3-smoke] launching catalogd (first start, empty catalog)...
|
|
[d3-smoke] POST /catalog/register (fresh):
|
|
✓ fresh register → existing=false, dataset_id=200a05a8-4f66-5a86-bdac-e17d87176613
|
|
[d3-smoke] GET /catalog/manifest/d3_smoke_dataset:
|
|
✓ manifest dataset_id matches
|
|
[d3-smoke] GET /catalog/list (1 entry):
|
|
✓ list count=1
|
|
[d3-smoke] restart catalogd → rehydrate from Parquet:
|
|
✓ rehydrated dataset_id matches across restart
|
|
[d3-smoke] re-register (same name + same fingerprint) → existing=true:
|
|
✓ existing=true, same dataset_id, objects replaced (count=2)
|
|
[d3-smoke] re-register (different fingerprint) → 409:
|
|
✓ different fingerprint → 409 Conflict
|
|
[d3-smoke] D3 acceptance gate: PASSED
|
|
[d3-smoke] cleanup
|
|
[d4-smoke] building storaged + catalogd + ingestd...
|
|
[d4-smoke] launching storaged → catalogd → ingestd...
|
|
[d4-smoke] POST /ingest?name=d4_workers (5 rows, 5 cols):
|
|
✓ ingest fresh → row_count=5, existing=false, key=datasets/d4_workers/247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet
|
|
[d4-smoke] mc shows the parquet on MinIO:
|
|
✓ 247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet present in lakehouse-go-primary/datasets/d4_workers/
|
|
[d4-smoke] catalogd manifest matches:
|
|
✓ manifest row_count=5, fp matches, 1 object at datasets/d4_workers/247165ad7d53e8d5993d3181dc9ce9b1d06383b336c31c999a89bd48d41308a4.parquet
|
|
[d4-smoke] ADR-010 — salary is string (mixed N/A):
|
|
✓ deferred to fingerprint stability (next test)
|
|
[d4-smoke] re-ingest same CSV → existing=true:
|
|
✓ idempotent re-ingest: existing=true, same dataset_id, same fingerprint
|
|
[d4-smoke] schema-drift CSV → 409:
|
|
✓ schema drift → 409 Conflict
|
|
[d4-smoke] D4 acceptance gate: PASSED
|
|
[d4-smoke] cleanup
|
|
[d5-smoke] building all 4 backing services...
|
|
[d5-smoke] launching storaged → catalogd → ingestd...
|
|
[d5-smoke] ingest 5-row CSV via D4 path:
|
|
✓ ingest row_count=5
|
|
[d5-smoke] launching queryd (initial Refresh picks up d5_workers)...
|
|
[d5-smoke] POST /sql SELECT count(*) FROM d5_workers:
|
|
✓ count(*)=5
|
|
[d5-smoke] POST /sql SELECT * FROM d5_workers LIMIT 3:
|
|
✓ rows[0] = (id=1, name=Alice), columns=[id, name, salary]
|
|
[d5-smoke] schema-drift ingest 409s; existing view still queries:
|
|
✓ drift → 409
|
|
✓ post-drift count(*)=5 (view unchanged)
|
|
[d5-smoke] error path: SELECT FROM nonexistent → 400:
|
|
✓ unknown table → 400
|
|
[d5-smoke] D5 acceptance gate: PASSED
|
|
[d5-smoke] cleanup
|
|
[d6-smoke] building all 5 binaries...
|
|
[d6-smoke] launching storaged → catalogd → ingestd...
|
|
[d6-smoke] launching gateway:
|
|
[d6-smoke] /v1/ingest?name=d6_workers (gateway → ingestd):
|
|
✓ ingest row_count=3, content-addressed key
|
|
[d6-smoke] /v1/catalog/list (gateway → catalogd):
|
|
✓ catalog count=1
|
|
[d6-smoke] /v1/storage/list?prefix=datasets/d6_workers/ (gateway → storaged):
|
|
✓ storage list returned 1 object(s) under datasets/d6_workers/
|
|
[d6-smoke] /v1/sql SELECT count(*) (gateway → queryd):
|
|
✓ count(*)=3
|
|
[d6-smoke] /v1/sql with row data (full round-trip):
|
|
✓ rows[0].name=Alice (full ingest → storage → catalog → query through gateway)
|
|
[d6-smoke] /v1/unknown → 404:
|
|
✓ unknown route → 404
|
|
[d6-smoke] D6 acceptance gate: PASSED
|
|
[d6-smoke] cleanup
|
|
[g1-smoke] building vectord + gateway...
|
|
[g1-smoke] launching vectord → gateway...
|
|
[g1-smoke] /v1/vectors/index — create dim=8:
|
|
✓ create → 201
|
|
[g1-smoke] duplicate create → 409:
|
|
✓ duplicate → 409
|
|
[g1-smoke] add batch of 200 vectors:
|
|
✓ added=200, length=200
|
|
[g1-smoke] search for inserted vector w-042 → recall:
|
|
✓ top hit = w-042 (dist=5.9604645E-8), 3 results, metadata round-tripped
|
|
[g1-smoke] dim mismatch on add → 400:
|
|
✓ dim mismatch → 400
|
|
[g1-smoke] search on missing index → 404:
|
|
✓ unknown index → 404
|
|
[g1-smoke] DELETE then GET → 404:
|
|
✓ post-delete GET → 404
|
|
[g1-smoke] G1 acceptance gate: PASSED
|
|
[g1-smoke] cleanup
|
|
[g1p-smoke] building storaged + vectord + gateway...
|
|
[g1p-smoke] launching storaged...
|
|
[g1p-smoke] launching vectord (round 1) → gateway...
|
|
[g1p-smoke] create index + add 50 vectors:
|
|
✓ added 50 → length=50
|
|
[g1p-smoke] verify storaged has the persistence file:
|
|
✓ _vectors/persist_demo.lhv1 present in storaged
|
|
[g1p-smoke] search pre-restart:
|
|
✓ pre-restart top hit = w-001
|
|
[g1p-smoke] kill + restart vectord (rehydrate path):
|
|
[g1p-smoke] vectord rehydrated index list shows persist_demo:
|
|
✓ list count=1 after restart
|
|
✓ length=50 after restart (state survived)
|
|
[g1p-smoke] search post-restart:
|
|
✓ post-restart top hit = w-001 (dist=0)
|
|
[g1p-smoke] DELETE then restart → index gone:
|
|
✓ persistence file removed from storaged
|
|
✓ post-delete restart list count=0
|
|
[g1p-smoke] G1P acceptance gate: PASSED
|
|
[g1p-smoke] cleanup
|
|
[g2-smoke] building embedd + vectord + gateway...
|
|
[g2-smoke] launching embedd → vectord (no persist) → gateway...
|
|
[g2-smoke] /v1/embed — two distinct texts:
|
|
✓ dim=768, model=nomic-embed-text-v2-moe, 2 distinct vectors
|
|
[g2-smoke] determinism — same text twice → byte-identical vector:
|
|
✓ identical text → identical vector
|
|
[g2-smoke] empty texts → 400:
|
|
✓ empty → 400
|
|
[g2-smoke] bad model → 502:
|
|
✓ unknown model → 502
|
|
[g2-smoke] end-to-end: embed → vectord add → search by embed → recall:
|
|
✓ embed → store → search round-trip: w-0 at dist=0
|
|
[g2-smoke] G2 acceptance gate: PASSED
|
|
[g2-smoke] cleanup
|
|
[chatd-smoke] building chatd + gateway...
|
|
[chatd-smoke] launching chatd → gateway...
|
|
[chatd-smoke] /v1/chat/providers — only ollama registered:
|
|
✓ exactly 1 provider (ollama, available=true)
|
|
[chatd-smoke] POST /v1/chat with bare model name:
|
|
✓ provider=ollama, latency=11134ms, content=ok…
|
|
[chatd-smoke] POST /v1/chat with explicit ollama/ prefix:
|
|
✓ ollama/qwen3.5:latest → provider=ollama, model=qwen3.5:latest (prefix stripped)
|
|
[chatd-smoke] POST /v1/chat with :cloud suffix (no cloud provider):
|
|
✓ kimi-k2.6:cloud → 404 (ollama_cloud not registered, no silent fall-through to local)
|
|
[chatd-smoke] POST /v1/chat with unknown/ prefix (falls through, upstream 502s):
|
|
✓ unknown/<model> → ollama default → upstream 502 (no silent prefix-strip)
|
|
[chatd-smoke] POST /v1/chat with missing model field:
|
|
✓ missing model → 400
|
|
[chatd-smoke] chatd acceptance gate: PASSED (6/6)
|
|
[chatd-smoke] cleanup
|
|
[downgrade-smoke] building matrixd + vectord + gateway...
|
|
[downgrade-smoke] launching vectord → matrixd → gateway...
|
|
[downgrade-smoke] strong model + no force → downgrade fires:
|
|
✓ codereview_lakehouse → codereview_isolation (downgraded_from=lakehouse)
|
|
[downgrade-smoke] forced_mode=true bypasses:
|
|
✓ caller-forced mode preserved, no downgrade
|
|
[downgrade-smoke] force_full_override=true bypasses:
|
|
✓ env-override bypass, no downgrade
|
|
[downgrade-smoke] weak model (qwen3.5:latest) bypasses:
|
|
✓ weak model keeps lakehouse
|
|
[downgrade-smoke] non-lakehouse mode → gate not applicable:
|
|
✓ codereview_isolation passes through unchanged
|
|
[downgrade-smoke] empty mode → 400:
|
|
✓ empty mode → 400
|
|
[downgrade-smoke] Downgrade gate acceptance: PASSED
|
|
[downgrade-smoke] cleanup
|
|
[matrix-smoke] building matrixd + vectord + gateway...
|
|
[matrix-smoke] launching vectord → matrixd → gateway...
|
|
[matrix-smoke] create two corpora:
|
|
✓ corpus_a and corpus_b created
|
|
[matrix-smoke] add vectors to both corpora:
|
|
✓ 3 + 3 vectors loaded
|
|
[matrix-smoke] /matrix/corpora lists both:
|
|
✓ count=2, both corpora listed
|
|
[matrix-smoke] /matrix/search multi-corpus retrieve+merge:
|
|
✓ 4 merged results · 3+3 per-corpus · both corpora represented
|
|
[matrix-smoke] top hit comes from corpus_b (b-near is globally closest):
|
|
✓ top hit: id=b-near corpus=corpus_b (closer than corpus_a's a-near)
|
|
[matrix-smoke] metadata preserved on merged results:
|
|
✓ metadata.label round-trips through matrix
|
|
[matrix-smoke] results sorted by distance ascending:
|
|
✓ distances ascending
|
|
[matrix-smoke] empty corpora → 400:
|
|
[matrix-smoke] missing corpus name → 502:
|
|
[matrix-smoke] no query (empty text and vector) → 400:
|
|
✓ empty=400, missing-corpus=502, no-query=400
|
|
[matrix-smoke] metadata_filter drops non-matching results:
|
|
✓ filter kept 2 ('a near' + 'b near'), dropped 4 mid/far entries
|
|
[matrix-smoke] Matrix acceptance gate: PASSED
|
|
[matrix-smoke] cleanup
|
|
[observer-smoke] building observerd + gateway...
|
|
[observer-smoke] launching observerd → gateway...
|
|
[observer-smoke] record 5 ops:
|
|
✓ 5 events posted
|
|
[observer-smoke] /observer/stats aggregates correctly:
|
|
✓ total=5 (3 ok + 2 fail) · by_source: mcp=3 scenario=2 · 2 scenario digests
|
|
[observer-smoke] empty endpoint → 400:
|
|
✓ empty endpoint rejected
|
|
[observer-smoke] kill + restart observerd → ops survive:
|
|
✓ total=5 ok=3 err=2 preserved through restart
|
|
[observer-smoke] Observer acceptance gate: PASSED
|
|
[observer-smoke] cleanup
|
|
[pathway-smoke] building pathwayd + gateway...
|
|
[pathway-smoke] launching pathwayd → gateway...
|
|
[pathway-smoke] Add → fresh UID + replay_count=1:
|
|
✓ uid=27f05e1f-4fee-4e8d-9409-9b7493ef9200 replay_count=1
|
|
[pathway-smoke] Get → returns same trace:
|
|
✓ content.approach round-trips
|
|
[pathway-smoke] AddIdempotent same UID → replay_count++:
|
|
✓ replay_count bumped to 2
|
|
[pathway-smoke] Update → in-place content replace:
|
|
✓ Update applied and persisted
|
|
[pathway-smoke] Revise → new UID with predecessor link:
|
|
✓ revision uid=9826a9d0-55f9-4fa7-b342-1bf692966d1a predecessor=27f05e1f-4fee-4e8d-9409-9b7493ef9200
|
|
[pathway-smoke] History → walks chain backward:
|
|
✓ chain length=2, [0]=9826a9d0-55f9-4fa7-b342-1bf692966d1a [1]=27f05e1f-4fee-4e8d-9409-9b7493ef9200
|
|
[pathway-smoke] Search tag=staffing → finds both traces:
|
|
✓ tag search count=2
|
|
[pathway-smoke] Retire → excluded from Search but Get-able:
|
|
✓ retired excluded from default Search, included with flag, still Get-able
|
|
[pathway-smoke] Stats → total/active/retired counters:
|
|
✓ total=2 active=1 retired=1
|
|
[pathway-smoke] Negative paths → 4xx semantics:
|
|
✓ get/update/revise/retire on unknown → 404; bad content → 400
|
|
[pathway-smoke] kill + restart pathwayd → state survives:
|
|
✓ replay_count, retired flag, predecessor link all preserved
|
|
[pathway-smoke] Pathway acceptance gate: PASSED
|
|
[pathway-smoke] cleanup
|
|
[playbook-smoke] building stack...
|
|
[playbook-smoke] launching embedd → vectord → matrixd → gateway...
|
|
[playbook-smoke] embedding 3 corpus items + query...
|
|
[playbook-smoke] create corpus widgets + add 3 items...
|
|
[playbook-smoke] baseline search (no playbook):
|
|
baseline order: widget-a,widget-b,widget-c widget-c distance=0.6565746
|
|
[playbook-smoke] record playbook: (alpha staffing query test full prompt) → widget-c score=1.0
|
|
✓ playbook_id=pb-4f1d0dccdb1df0ae
|
|
[playbook-smoke] boosted search (use_playbook=true):
|
|
boosted order: widget-a,widget-c,widget-b widget-c distance=0.3282873 playbook_boosted=1
|
|
✓ playbook_boosted=1 ≥ 1
|
|
widget-c distance ratio (boosted/baseline) = 0.5 (expect ≈ 0.5)
|
|
✓ ratio in [0.40, 0.60] — boost applied correctly
|
|
[playbook-smoke] bulk record 3 entries:
|
|
✓ 2 recorded, 1 failed (empty query_text caught), per-entry IDs/errors returned
|
|
[playbook-smoke] Playbook acceptance gate: PASSED
|
|
[playbook-smoke] cleanup
|
|
[relevance-smoke] building matrixd + vectord + gateway...
|
|
[relevance-smoke] launching vectord → matrixd → gateway...
|
|
[relevance-smoke] adjacency-pollution: Connector outranks Registry, junk dropped:
|
|
✓ Connector kept, junk dropped, Connector (0.6799999999999999) > Registry (-0.45555555555555555)
|
|
[relevance-smoke] empty chunks → 400:
|
|
✓ 400 on empty chunks
|
|
[relevance-smoke] threshold=10 (impossibly high) drops everything:
|
|
✓ threshold=10 drops everything (0 kept / 1 dropped)
|
|
[relevance-smoke] Relevance acceptance gate: PASSED
|
|
[relevance-smoke] cleanup
|
|
[cap-smoke] building storaged + gateway...
|
|
[cap-smoke] launching storaged → gateway...
|
|
[cap-smoke] generating 300 MiB deterministic payload...
|
|
size=314572800 sha=17a88af83717...
|
|
[cap-smoke] Test 1: PUT 300 MiB to _vectors/ (should pass)
|
|
✓ PUT _vectors/ → 200
|
|
[cap-smoke] Test 2: PUT 300 MiB to datasets/ (should reject)
|
|
✓ PUT datasets/ → 413 (default cap protects routine prefixes)
|
|
[cap-smoke] Test 3: GET _vectors/ — sha matches input
|
|
✓ GET round-trip preserves bytes (size=314572800 sha=17a88af83717)
|
|
[cap-smoke] ✓ Storaged cap smoke: PASSED
|
|
[cap-smoke] cleanup
|
|
[workflow-smoke] building observerd + gateway...
|
|
[workflow-smoke] launching observerd → gateway...
|
|
[workflow-smoke] /observer/workflow/modes lists fixtures + real modes:
|
|
✓ all 7 expected modes registered (fixtures + 4 pure + matrix.search HTTP)
|
|
[workflow-smoke] 3-node DAG: shape (upper) → weakness → improvement
|
|
✓ status=succeeded · shape=HELLO WORLD · refs propagated through 3-node chain
|
|
[workflow-smoke] /observer/stats reflects workflow ops:
|
|
✓ 3 workflow ops recorded (one per node), total=3
|
|
[workflow-smoke] unknown mode → 400:
|
|
✓ unknown mode aborts with 400 + helpful error
|
|
[workflow-smoke] real-mode chain: downgrade → distillation.score
|
|
✓ downgrade flipped lakehouse→isolation; scorer rated scrum_review attempt_1=accepted
|
|
[workflow-smoke] Workflow runner acceptance: PASSED
|
|
[workflow-smoke] cleanup
|
|
[materializer-smoke] building bin/materializer...
|
|
[materializer-smoke] dry-run probe
|
|
[materializer-smoke] first run
|
|
[evidence_index] 4 read · 3 written · 1 skipped · 0 deduped
|
|
data/_kb/distilled_facts.jsonl: read=3 wrote=2 skip=1 dedup=0
|
|
data/_kb/distilled_procedures.jsonl: (missing — skipped)
|
|
data/_kb/distilled_config_hints.jsonl: (missing — skipped)
|
|
data/_kb/contract_analyses.jsonl: (missing — skipped)
|
|
data/_kb/mode_experiments.jsonl: (missing — skipped)
|
|
data/_kb/scrum_reviews.jsonl: (missing — skipped)
|
|
data/_kb/observer_escalations.jsonl: read=1 wrote=1 skip=0 dedup=0
|
|
data/_kb/audit_facts.jsonl: (missing — skipped)
|
|
data/_kb/auto_apply.jsonl: (missing — skipped)
|
|
data/_kb/observer_reviews.jsonl: (missing — skipped)
|
|
data/_kb/audits.jsonl: (missing — skipped)
|
|
data/_kb/outcomes.jsonl: (missing — skipped)
|
|
[evidence_index] receipt: /tmp/tmp.eOKwqXIezb/reports/distillation/2026-05-02T08-54-40-881776326Z/receipt.json
|
|
[evidence_index] validation_pass=false
|
|
[materializer-smoke] idempotent re-run
|
|
[materializer-smoke] PASS
|
|
[replay-smoke] building bin/replay...
|
|
[replay-smoke] dry-run (with retrieval)
|
|
[replay-smoke] dry-run (no retrieval)
|
|
[replay-smoke] forced-fail with escalation
|
|
[replay-smoke] PASS
|
|
[validatord-smoke] building validatord + gateway...
|
|
[validatord-smoke] launching validatord → gateway...
|
|
✓ validatord roster loaded with 3 records
|
|
[validatord-smoke] /v1/validate playbook happy path:
|
|
✓ playbook OK ({"findings":[],"elapsed_ms":0})
|
|
[validatord-smoke] /v1/validate playbook missing fingerprint → 422:
|
|
✓ playbook missing fingerprint → 422 schema/fingerprint
|
|
[validatord-smoke] /v1/validate fill with phantom candidate → 422:
|
|
✓ phantom candidate W-PHANTOM → 422 consistency
|
|
[validatord-smoke] /v1/validate unknown kind → 400:
|
|
✓ unknown kind → 400
|
|
[validatord-smoke] PASS — 5/5 probes through gateway :3110
|
|
[validatord-smoke] cleanup
|