golangLAKEHOUSE

profit/golangLAKEHOUSE

Fork 0

Commit Graph

Author	SHA1	Message	Date
root	6af0520ed2	A: fail-loud on non-loopback bind — closes worst case of R-001 shared.Run now refuses to bind a non-loopback address unless the LH_<SERVICE>_ALLOW_NONLOOPBACK=1 env is set. Single change covers all 7 binaries via the existing Run call site; no per-binary wiring needed. Closes the accidental-0.0.0.0 deploy attack surface for R-001: queryd /sql is RCE-equivalent off loopback (DuckDB has filesystem read + COPY TO + read_text), but the gate applies to every binary uniformly so the same posture covers vectord (mutation routes), catalogd (manifest writes), and the others. What passes the gate: 127.0.0.1:port, 127.x.y.z:port (full /8), [::1]:port, localhost:port, OR explicit env LH_<SVC>_ALLOW_NONLOOPBACK=1 What fail-louds: 0.0.0.0:port, [::]:port, :port (all interfaces), any non-loopback IP, any non-localhost hostname, unparseable shapes ("", "no port", garbage) Override env is strict equality "1" — typos like "true"/"yes" do NOT trigger it, so a future operator can't accidentally expose by typing the wrong value. Override fires log a structured warn so the choice is auditable in production. Error message cites the env name AND R-001 by name so operators see the fix path without grepping: "refusing non-loopback bind \"0.0.0.0:3214\" for \"queryd\" (set LH_QUERYD_ALLOW_NONLOOPBACK=1 to override; see audit R-001)" internal/shared/bind.go — requireLoopbackOrOverride + isLoopbackAddr internal/shared/bind_test.go — 7 test funcs incl. table-driven IPv4/IPv6/hostname coverage and per-service env isolation internal/shared/server.go — 1-line gate in Run before listen Verified: go test -short ./internal/shared/ — all green (was 14 funcs, now 21) just verify — vet + test + 9 smokes still 33s Doesn't address R-001's full attack surface (any reachable port can issue arbitrary SQL); ADR-003 + Bearer-token middleware is the follow-up. This commit makes the implicit "localhost-only is the auth layer" guarantee explicit and un-bypassable without explicit env. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-29 05:56:42 -05:00
Claw	ad2ec1aca9	G0 D1 hardened: 3-lineage scrum review on shipped code · 7 fixes applied Code-review pass after D1 shipped, all three model lineages running in parallel against the actual Go source (not docs): Convergent findings (≥2 reviewers — high confidence): - C1 BLOCK · Run() errCh/select race could silently drop fast bind errors. Fixed: net.Listen() now runs synchronously before the goroutine; bind errors surface as Run()'s return value. - C2 BLOCK · scripts/d1_smoke.sh sleep 0.5 races bind on cold boxes. Fixed: replaced with poll_health() loop, 5s/svc budget, 50ms poll. - C3 WARN · LoadConfig silent fallback when file missing. Fixed: emits slog.Warn with path + hint when path given but file absent. Single-reviewer fixes: - S1 WARN · slog.SetDefault inside Run() mutated global state from a library function. Fixed: Run() no longer calls SetDefault. - S2 WARN · os.IsNotExist → errors.Is(err, fs.ErrNotExist) idiom. - S6 WARN · smoke double-curl collapsed to single curl -i parse. Second-pass Opus review on post-fix code caught one more: - head -1 on curl -i fragile against 1xx interim lines. Fixed: awk picks the last HTTP/* status line (robust to 100 Continue). Accepted with rationale (deferred or planned): - S3 secrets-in-lakehouse.toml: D2.3 SecretsProvider already planned - S4 5x cmd/*/main.go duplication: defer until D2 reveals real per-service config consumption - S5 /health log volume: defer post-G0, not on k8s yet - 2nd-pass theoreticals: clean-exit-no-Shutdown path doesn't trigger, defensive defer ln.Close() aspirational, etc. Verification: - go build ./cmd/... exit 0 - go vet ./... clean - ./scripts/d1_smoke.sh D1 acceptance gate: PASSED - 3-lineage code review · 14 findings · 7 fixed · 0 deferred · 5 accepted with rationale Total D1 review coverage across the phase: - 3 doc-review passes (Opus + Kimi + Qwen) — 13 findings, 10 fixed - 1 runtime smoke — 1 finding (port 3100 collision), fixed - 1 code-review parallel pass — 14 findings, 7 fixed - 1 code-review second pass (Opus) — 1 actionable, fixed - Cumulative: 29 findings · 19 fixed inline · 5 accepted · 5 deferred Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-28 07:07:50 -05:00
Claw	1142f54f23	G0 D1 ships: skeleton + chi + /health × 5 binaries · acceptance gate PASSED Phase G0 Day 1 executed end-to-end after a third-pass review by qwen3-coder:480b consolidated all findings across Opus/Kimi/Qwen lineages. Cross-lineage review consolidation (3 model passes + 1 runtime pass): - Opus 4.7: 9 findings · 7 fixed inline · 2 deferred - Kimi K2.6: 2 BLOCKs (introduced by Opus fixes) · 2 fixed - Qwen3-coder:480b: 2 WARNs · 1 fixed (D2.4 256 MiB cap + 4-slot semaphore on PUTs) · 1 deferred (Q2 view refresh batching) - Runtime smoke: 1 finding (port 3100 collision with live Rust lakehouse) · fixed (Go dev ports shifted to 3110+) - Total: 14 findings · 11 fixed · 3 deferred to G2 What landed in code: - internal/shared/server.go — chi factory, slog JSON, /health, graceful shutdown via signal.NotifyContext - internal/shared/config.go — TOML loader, DefaultConfig, -config flag - cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go — five binaries, each ~30 lines using the shared factory - lakehouse.toml — G0 dev defaults (3110-3214) - scripts/d1_smoke.sh — repeatable smoke that exits 0 on PASS - go.mod / go.sum — chi v5.2.5, pelletier/go-toml/v2 v2.3.0 Verified end-to-end via scripts/d1_smoke.sh: - All 5 /health endpoints return 200 with correct service name - Gateway /v1/ingest + /v1/sql stubs return 501 with X-Lakehouse-Stub - Graceful shutdown logs cleanly on SIGTERM - DuckDB cgo path verified separately (sql.Open("duckdb","") + ping) D1 ACCEPTANCE GATE: PASSED. Next: D2 — storaged S3 GET/PUT/LIST against MinIO. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-28 07:00:37 -05:00

Author

SHA1

Message

Date

root

6af0520ed2

A: fail-loud on non-loopback bind — closes worst case of R-001

shared.Run now refuses to bind a non-loopback address unless the
LH_<SERVICE>_ALLOW_NONLOOPBACK=1 env is set. Single change covers
all 7 binaries via the existing Run call site; no per-binary
wiring needed.

Closes the accidental-0.0.0.0 deploy attack surface for R-001:
queryd /sql is RCE-equivalent off loopback (DuckDB has filesystem
read + COPY TO + read_text), but the gate applies to every binary
uniformly so the same posture covers vectord (mutation routes),
catalogd (manifest writes), and the others.

What passes the gate:
  127.0.0.1:port, 127.x.y.z:port (full /8), [::1]:port,
  localhost:port, OR explicit env LH_<SVC>_ALLOW_NONLOOPBACK=1

What fail-louds:
  0.0.0.0:port, [::]:port, :port (all interfaces),
  any non-loopback IP, any non-localhost hostname,
  unparseable shapes ("", "no port", garbage)

Override env is strict equality "1" — typos like "true"/"yes" do NOT
trigger it, so a future operator can't accidentally expose by typing
the wrong value. Override fires log a structured warn so the choice
is auditable in production.

Error message cites the env name AND R-001 by name so operators see
the fix path without grepping:
  "refusing non-loopback bind \"0.0.0.0:3214\" for \"queryd\"
   (set LH_QUERYD_ALLOW_NONLOOPBACK=1 to override; see audit R-001)"

internal/shared/bind.go            — requireLoopbackOrOverride + isLoopbackAddr
internal/shared/bind_test.go       — 7 test funcs incl. table-driven
                                     IPv4/IPv6/hostname coverage and
                                     per-service env isolation
internal/shared/server.go          — 1-line gate in Run before listen

Verified:
  go test -short ./internal/shared/ — all green (was 14 funcs, now 21)
  just verify                       — vet + test + 9 smokes still 33s

Doesn't address R-001's full attack surface (any reachable port can
issue arbitrary SQL); ADR-003 + Bearer-token middleware is the
follow-up. This commit makes the implicit "localhost-only is the auth
layer" guarantee explicit and un-bypassable without explicit env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-29 05:56:42 -05:00

Claw

ad2ec1aca9

G0 D1 hardened: 3-lineage scrum review on shipped code · 7 fixes applied

Code-review pass after D1 shipped, all three model lineages running
in parallel against the actual Go source (not docs):

Convergent findings (≥2 reviewers — high confidence):
- C1 BLOCK · Run() errCh/select race could silently drop fast bind
  errors. Fixed: net.Listen() now runs synchronously before the
  goroutine; bind errors surface as Run()'s return value.
- C2 BLOCK · scripts/d1_smoke.sh sleep 0.5 races bind on cold boxes.
  Fixed: replaced with poll_health() loop, 5s/svc budget, 50ms poll.
- C3 WARN · LoadConfig silent fallback when file missing. Fixed:
  emits slog.Warn with path + hint when path given but file absent.

Single-reviewer fixes:
- S1 WARN · slog.SetDefault inside Run() mutated global state from a
  library function. Fixed: Run() no longer calls SetDefault.
- S2 WARN · os.IsNotExist → errors.Is(err, fs.ErrNotExist) idiom.
- S6 WARN · smoke double-curl collapsed to single curl -i parse.

Second-pass Opus review on post-fix code caught one more:
- head -1 on curl -i fragile against 1xx interim lines. Fixed:
  awk picks the last HTTP/* status line (robust to 100 Continue).

Accepted with rationale (deferred or planned):
- S3 secrets-in-lakehouse.toml: D2.3 SecretsProvider already planned
- S4 5x cmd/*/main.go duplication: defer until D2 reveals real
  per-service config consumption
- S5 /health log volume: defer post-G0, not on k8s yet
- 2nd-pass theoreticals: clean-exit-no-Shutdown path doesn't trigger,
  defensive defer ln.Close() aspirational, etc.

Verification:
- go build ./cmd/...  exit 0
- go vet ./...         clean
- ./scripts/d1_smoke.sh  D1 acceptance gate: PASSED
- 3-lineage code review · 14 findings · 7 fixed · 0 deferred · 5
  accepted with rationale

Total D1 review coverage across the phase:
- 3 doc-review passes (Opus + Kimi + Qwen) — 13 findings, 10 fixed
- 1 runtime smoke — 1 finding (port 3100 collision), fixed
- 1 code-review parallel pass — 14 findings, 7 fixed
- 1 code-review second pass (Opus) — 1 actionable, fixed
- Cumulative: 29 findings · 19 fixed inline · 5 accepted · 5 deferred

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-28 07:07:50 -05:00

Claw

1142f54f23

G0 D1 ships: skeleton + chi + /health × 5 binaries · acceptance gate PASSED

Phase G0 Day 1 executed end-to-end after a third-pass review by
qwen3-coder:480b consolidated all findings across Opus/Kimi/Qwen
lineages.

Cross-lineage review consolidation (3 model passes + 1 runtime pass):
- Opus 4.7: 9 findings · 7 fixed inline · 2 deferred
- Kimi K2.6: 2 BLOCKs (introduced by Opus fixes) · 2 fixed
- Qwen3-coder:480b: 2 WARNs · 1 fixed (D2.4 256 MiB cap + 4-slot
  semaphore on PUTs) · 1 deferred (Q2 view refresh batching)
- Runtime smoke: 1 finding (port 3100 collision with live Rust
  lakehouse) · fixed (Go dev ports shifted to 3110+)
- Total: 14 findings · 11 fixed · 3 deferred to G2

What landed in code:
- internal/shared/server.go — chi factory, slog JSON, /health,
  graceful shutdown via signal.NotifyContext
- internal/shared/config.go — TOML loader, DefaultConfig, -config flag
- cmd/{gateway,storaged,catalogd,ingestd,queryd}/main.go — five
  binaries, each ~30 lines using the shared factory
- lakehouse.toml — G0 dev defaults (3110-3214)
- scripts/d1_smoke.sh — repeatable smoke that exits 0 on PASS
- go.mod / go.sum — chi v5.2.5, pelletier/go-toml/v2 v2.3.0

Verified end-to-end via scripts/d1_smoke.sh:
- All 5 /health endpoints return 200 with correct service name
- Gateway /v1/ingest + /v1/sql stubs return 501 with X-Lakehouse-Stub
- Graceful shutdown logs cleanly on SIGTERM
- DuckDB cgo path verified separately (sql.Open("duckdb","") + ping)

D1 ACCEPTANCE GATE: PASSED.

Next: D2 — storaged S3 GET/PUT/LIST against MinIO.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-28 07:00:37 -05:00

3 Commits