Implements the auth posture from ADR-003 (commit 0d18ffa). Two
independent layers — Bearer token (constant-time compare via
crypto/subtle) and IP allowlist (CIDR set) — composed in shared.Run
so every binary inherits the same gate without per-binary wiring.
Together with the bind-gate from commit 6af0520, this mechanically
closes audit risks R-001 + R-007:
- non-loopback bind without auth.token = startup refuse
- non-loopback bind WITH auth.token + override env = allowed
- loopback bind = all gates open (G0 dev unchanged)
internal/shared/auth.go (NEW)
RequireAuth(cfg AuthConfig) returns chi-compatible middleware.
Empty Token + empty AllowedIPs → pass-through (G0 dev mode).
Token-only → 401 Bearer mismatch.
AllowedIPs-only → 403 source IP not in CIDR set.
Both → both gates apply.
/health bypasses both layers (load-balancer / liveness probes
shouldn't carry tokens).
CIDR parsing pre-runs at boot; bare IP (no /N) treated as /32 (or
/128 for IPv6). Invalid entries log warn and drop, fail-loud-but-
not-fatal so a typo doesn't kill the binary.
Token comparison: subtle.ConstantTimeCompare on the full
"Bearer <token>" wire-format string. Length-mismatch returns 0
(per stdlib spec), so wrong-length tokens reject without timing
leak. Pre-encoded comparison slice stored in the middleware
closure — one allocation per request.
Source-IP extraction prefers net.SplitHostPort fallback to
RemoteAddr-as-is for httptest compatibility. X-Forwarded-For
support is a follow-up when a trusted proxy fronts the gateway
(config knob TBD per ADR-003 §"Future").
internal/shared/server.go
Run signature: gained AuthConfig parameter (4th arg).
/health stays mounted on the outer router (public).
Registered routes go inside chi.Group with RequireAuth applied —
empty config = transparent group.
Added requireAuthOnNonLoopback startup check: non-loopback bind
with empty Token = refuse to start (cites R-001 + R-007 by name).
internal/shared/config.go
AuthConfig type added with TOML tags. Fields: Token, AllowedIPs.
Composed into Config under [auth].
cmd/<svc>/main.go × 7 (catalogd, embedd, gateway, ingestd, queryd,
storaged, vectord, mcpd is unaffected — stdio doesn't bind a port)
Each call site adds cfg.Auth as the 4th arg to shared.Run. No
other changes — middleware applies via shared.Run uniformly.
internal/shared/auth_test.go (12 test funcs)
Empty config pass-through, missing-token 401, wrong-token 401,
correct-token 200, raw-token-without-Bearer-prefix 401, /health
always public, IP allowlist allow + reject, bare IP /32, both
layers when both configured, invalid CIDR drop-with-warn, RemoteAddr
shape extraction. The constant-time comparison is verified by
inspection (comments in auth.go) plus the existence of the
passthrough test (length-mismatch case).
Verified:
go test -count=1 ./internal/shared/ — all green (was 21, now 33 funcs)
just verify — vet + test + 9 smokes 33s
just proof contract — 53/0/1 unchanged
Smokes + proof harness keep working without any token configuration:
default Auth is empty struct → middleware is no-op → existing tests
pass unchanged. To exercise the gate, operators set [auth].token in
lakehouse.toml (or, per the "future" note in the ADR, via env var).
Closes audit findings:
R-001 HIGH — fully mechanically closed (was: partial via bind gate)
R-007 MED — fully mechanically closed (was: design-only ADR-003)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds main_test.go for each of the 6 cmd binaries that lacked them
(storaged already had main_test.go; that's where the pattern came
from). Each test file focuses on the cmd-specific surface — route
mounts, body caps, decode/validation paths — without re-testing
internal package logic that's covered elsewhere.
cmd/catalogd/main_test.go — 6 funcs
TestRoutesMounted: chi.Walk asserts /catalog/{register,manifest/*,list}
TestHandleRegister_BodyTooLarge: 5 MiB body → 4xx
TestHandleRegister_MalformedJSON: 400
TestHandleRegister_EmptyName_400: ErrEmptyName surfaces as 400
TestHandleGetManifest_404 + TestHandleList_EmptyShape
cmd/embedd/main_test.go — 8 funcs
stubProvider implements embed.Provider deterministically
TestRoutesMounted, MalformedJSON_400, EmptyTextRejected_400 (per
scrum O-W3), UpstreamError_502 (provider error → 502, not 500),
HappyPath_ProviderEcho, BodyTooLarge (4xx range), TestItoa
(covers the no-strconv helper)
cmd/gateway/main_test.go — 4 funcs
TestMustParseUpstream_HappyPaths: 3 valid URLs
TestMustParseUpstream_FailureExits: re-execs the test binary in a
subprocess with env flag (standard pattern for testing os.Exit
callers); subprocess invokes mustParseUpstream("127.0.0.1:3211")
[missing scheme]; expects exit non-zero. Same pattern for garbage.
TestUpstreamConfigKeys_DocumentedShape: locks the 6 _url keys
cmd/ingestd/main_test.go — 7 funcs
Stubs both storaged and catalogd via httptest.Server so the cmd
layer can be exercised without bringing the full chain up.
TestHandleIngest_MissingNameQueryParam: 400 with "name" in body
TestHandleIngest_MalformedMultipart: 400
TestHandleIngest_MissingFormFile: 400 (valid multipart, wrong field)
TestHandleIngest_BodyTooLarge: 4xx
TestEscapeKeyPath: 6-case URL-escape table (apostrophe, space, etc.)
TestParquetKeyPath_Format: locks the datasets/<n>/<fp>.parquet shape
per scrum C-DRIFT (any rename breaks idempotent re-ingest)
cmd/queryd/main_test.go — 6 funcs
Tests pre-DB paths (decode, body cap, empty SQL); db.QueryContext
itself needs DuckDB so it's covered by GOLAKE-040 in the proof
harness, not unit tests. handlers.db = nil here is intentional.
TestHandleSQL_EmptySQL_400: 3 cases (empty, whitespace, mixed-WS)
TestMaxSQLBodyBytes_Reasonable: locks the 64 KiB constant in a
sane range so a refactor can't blow it open
TestPrimaryBucket_Constant: locks "primary" — secrets lookup uses
this; rename = silent secret-resolution failure at boot
cmd/vectord/main_test.go — 14 funcs
All 6 routes verified mounted. handlers.persist = nil = pure
in-memory mode; persistence is GOLAKE-070 in the proof harness.
Coverage of every error branch in handleCreate/Add/Search/Delete:
missing index → 404, dim mismatch → 400, empty items → 400,
empty id → 400, malformed JSON → 400, body too large → 4xx,
happy create → 201, happy list → 200.
One real finding caught during writing:
Body-cap rejection is sometimes 413 (typed MaxBytesError survives
unwrap) and sometimes 400 (decoder wraps it as a generic decode
error). Both are valid client-error contracts; the contract isn't
"exactly 413" but "fails loud as 4xx, never silent 200 or 5xx."
Tests assert 4xx range. The proof harness's
proof_assert_status_4xx already had this shape — just bringing
the unit tests in line with it.
Verified:
go test -count=1 -short ./cmd/... — all 7 packages green
just verify — vet + test + 9 smokes 35s
Closes audit risk R-005 (6/7 cmd/main.go untested). Combined with
the proof harness's wiring coverage, every cmd-level handler now
has both unit-test and integration-test coverage of the wiring
layer. R-005 → CLOSED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds optional persistence to vectord (G1's HNSW vector search). Single-
file framed format per index — eliminates the torn-write class that
the 3-way convergent scrum finding identified:
_vectors/<name>.lhv1 — single binary blob:
[4 bytes magic "LHV1"]
[4 bytes envelope_len uint32 BE]
[envelope bytes — JSON params + metadata + version]
[graph bytes — raw hnsw.Graph.Export]
Pre-extraction: internal/catalogd/store_client.go → internal/storeclient/
shared package, since both catalogd and vectord need it. Same pattern as
the pre-D5 catalogclient extraction.
Optional via [vectord].storaged_url config (empty = ephemeral mode).
On startup: List + Load each persisted index. After Create / batch Add /
DELETE: Save (or Delete from storaged). Save failures are logged-not-
fatal — in-memory state is the source of truth in flight.
Acceptance smoke G1P 8/8 PASS — kill+restart preserves state, post-
restart search returns dist=0 (graph round-trips exactly), DELETE
removes the file, post-delete restart shows count=0.
All 8 smokes (D1-D6 + G1 + G1P) PASS deterministically. The g1_smoke
gained scripts/g1_smoke.toml that disables persistence so the
in-memory API test stays decoupled from any rehydrate-from-storaged
state contamination.
Cross-lineage scrum on shipped code:
- Opus 4.7 (opencode): 1 BLOCK + 5 WARN + 3 INFO
- Kimi K2-0905 (openrouter): 1 BLOCK + 2 WARN
- Qwen3-coder (openrouter): 2 BLOCK + 2 WARN + 1 INFO
Fixed (3 — 1 convergent + 2 single-reviewer):
C1 (Opus + Kimi + Qwen 3-WAY CONVERGENT WARN): Save was non-atomic
across two PUTs — envelope-succeeds + graph-fails left a half-
saved index that passed the "both present" List filter and
silently mismatched metadata against vectors on Load.
Fix: collapse to single framed file (no torn-write window
possible).
O-B1 (Opus BLOCK): isNotFound substring-matched "key not found"
against the wrapped error message — brittle, any 5xx body
containing that text would silently misclassify as missing.
Fix: errors.Is(err, storeclient.ErrKeyNotFound).
O-I3 (Opus INFO): handleAdd pre-validation only covered id+dim;
NaN/Inf/zero-norm could still fail mid-batch leaving partial
commits. Fix: extend pre-validation to call ValidateVector
(newly exported) per item before any commit.
Dismissed (3 false positives):
K-B1 + Q-B1 ("safeKey double-escapes %2F segments") — false
convergent. Wire-protocol escape is decoded by storaged's chi
router on the way in; on-disk key is the original literal.
%2F round-trips correctly through PathEscape → URL → chi decode
→ S3 key.
Q-B2 ("List vulnerable to race conditions") — vectord is single-
process; no concurrent Save against List in the same vectord.
Deferred (3): rehydrate per-index timeout (G2+ multi-index scale),
saveAfter request ctx (matches G0 timeout deferral), Encode RLock
during slow writer (documented as buffer-only API).
The C1 finding is the strongest signal of the cross-lineage filter:
three independent reviewers all flagged the same torn-write hazard.
Single-file framing eliminates the class — there's now no Persistor
state where envelope and graph can disagree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase G0 Day 3 ships catalogd: Arrow Parquet manifest codec, in-memory
registry with the ADR-020 idempotency contract (same name+fingerprint
reuses dataset_id; different fingerprint → 409 Conflict), HTTP client
to storaged for persistence, and rehydration on startup. Acceptance
smoke 6/6 PASSES end-to-end including rehydrate-across-restart — the
load-bearing test that the catalog/storaged service split actually
preserves state.
dataset_id derivation diverges from Rust: UUIDv5(namespace, name)
instead of v4 surrogate. Same name on any box generates the same
dataset_id; rehydrate after disk loss converges to the same identity
rather than silently re-issuing. Namespace pinned at
a8f3c1d2-4e5b-5a6c-9d8e-7f0a1b2c3d4e — every dataset_id ever issued
depends on these bytes.
Cross-lineage scrum on shipped code:
- Opus 4.7 (opencode): 1 BLOCK + 5 WARN + 3 INFO
- Kimi K2-0905 (openrouter, validated D2): 2 BLOCK + 2 WARN + 1 INFO
- Qwen3-coder (openrouter): 2 BLOCK + 2 WARN + 2 INFO
Fixed:
C1 list-offsets BLOCK (3-way convergent) → ValueOffsets(0) + bounds
C2 Rehydrate mutex held across I/O → swap-under-brief-lock pattern
S1 split-brain on persist failure → candidate-then-swap
S2 brittle string-match for 400 vs 500 → ErrEmptyName/ErrEmptyFingerprint sentinels
S3 Get/List shallow-copy aliasing → cloneManifest deep copy
S4 keep-alive socket leak on error paths → drainAndClose helper
Dismissed (false positives, all single-reviewer):
Kimi BLOCK "Decode crashes on empty Parquet" — already handled
Kimi INFO "safeKey double-escapes" — wrong, splitting before escape is required
Qwen INFO "rb.NewRecord() error unchecked" — API returns no error
Deferred to G1+: name validation regex, per-call deadlines, Snappy
compression, list pagination continuation tokens (storaged caps at
10k with sentinel for now).
Build clean, vet clean, all tests pass, smoke 6/6 PASS after every
fix round. arrow-go/v18 + google/uuid added; Go 1.24 → 1.25 forced
by arrow-go's minimum.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
root