66a704ca3e
Phase G0 Day 3 ships catalogd: Arrow Parquet manifest codec, in-memory registry with the ADR-020 idempotency contract (same name+fingerprint reuses dataset_id; different fingerprint → 409 Conflict), HTTP client to storaged for persistence, and rehydration on startup. Acceptance smoke 6/6 PASSES end-to-end including rehydrate-across-restart — the load-bearing test that the catalog/storaged service split actually preserves state. dataset_id derivation diverges from Rust: UUIDv5(namespace, name) instead of v4 surrogate. Same name on any box generates the same dataset_id; rehydrate after disk loss converges to the same identity rather than silently re-issuing. Namespace pinned at a8f3c1d2-4e5b-5a6c-9d8e-7f0a1b2c3d4e — every dataset_id ever issued depends on these bytes. Cross-lineage scrum on shipped code: - Opus 4.7 (opencode): 1 BLOCK + 5 WARN + 3 INFO - Kimi K2-0905 (openrouter, validated D2): 2 BLOCK + 2 WARN + 1 INFO - Qwen3-coder (openrouter): 2 BLOCK + 2 WARN + 2 INFO Fixed: C1 list-offsets BLOCK (3-way convergent) → ValueOffsets(0) + bounds C2 Rehydrate mutex held across I/O → swap-under-brief-lock pattern S1 split-brain on persist failure → candidate-then-swap S2 brittle string-match for 400 vs 500 → ErrEmptyName/ErrEmptyFingerprint sentinels S3 Get/List shallow-copy aliasing → cloneManifest deep copy S4 keep-alive socket leak on error paths → drainAndClose helper Dismissed (false positives, all single-reviewer): Kimi BLOCK "Decode crashes on empty Parquet" — already handled Kimi INFO "safeKey double-escapes" — wrong, splitting before escape is required Qwen INFO "rb.NewRecord() error unchecked" — API returns no error Deferred to G1+: name validation regex, per-call deadlines, Snappy compression, list pagination continuation tokens (storaged caps at 10k with sentinel for now). Build clean, vet clean, all tests pass, smoke 6/6 PASS after every fix round. arrow-go/v18 + google/uuid added; Go 1.24 → 1.25 forced by arrow-go's minimum. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
33 lines
921 B
TOML
33 lines
921 B
TOML
# Lakehouse-Go config — G0 dev defaults. Overrides via env are a
|
|
# G1+ concern; for G0 edit this file and restart the affected service.
|
|
|
|
# G0 dev ports — shifted to 3110+ so the Go services run alongside
|
|
# the live Rust lakehouse on 3100/3201-3204 without colliding. G5
|
|
# (demo cutover) flips gateway back to 3100 when Rust retires.
|
|
[gateway]
|
|
bind = "127.0.0.1:3110"
|
|
|
|
[storaged]
|
|
bind = "127.0.0.1:3211"
|
|
|
|
[catalogd]
|
|
bind = "127.0.0.1:3212"
|
|
storaged_url = "http://127.0.0.1:3211"
|
|
|
|
[ingestd]
|
|
bind = "127.0.0.1:3213"
|
|
|
|
[queryd]
|
|
bind = "127.0.0.1:3214"
|
|
|
|
[s3]
|
|
endpoint = "http://localhost:9000"
|
|
region = "us-east-1"
|
|
bucket = "lakehouse-go-primary" # G0 dedicated bucket so Rust + Go coexist
|
|
access_key_id = "" # populated by SecretsProvider from /etc/lakehouse/secrets-go.toml
|
|
secret_access_key = "" # ditto
|
|
use_path_style = true
|
|
|
|
[log]
|
|
level = "info"
|