b1d52306ad
Last day of Phase G0. Gateway promotes the D1 stub endpoints into
real reverse-proxies on :3110 fronting storaged + catalogd + ingestd
+ queryd. /v1 prefix lives at the edge — internal services route on
/storage, /catalog, /ingest, /sql, with the prefix stripped by a
custom Director per Kimi K2's D1-plan finding.
Routes:
/v1/storage/* → storaged
/v1/catalog/* → catalogd
/v1/ingest → ingestd
/v1/sql → queryd
Acceptance smoke 6/6 PASS — every assertion goes through :3110, none
direct to backing services. Full ingest → storage → catalog → query
round-trip verified end-to-end. The smoke's "rows[0].name=Alice"
assertion is the architectural payoff: five binaries, six HTTP
routes, one round-trip through one edge.
Cross-lineage scrum on shipped code:
- Opus 4.7 (opencode): 1 BLOCK + 2 WARN + 2 INFO
- Kimi K2-0905 (openrouter): 1 BLOCK + 3 WARN + 1 INFO (3 false positives, all from one wrong TrimPrefix theory)
- Qwen3-coder (openrouter): 5 completion tokens — "No BLOCKs."
Fixed (2, both Opus single-reviewer):
O-BLOCK: Director path stripping fails if upstream URL has a
non-empty path. The default Director's singleJoiningSlash runs
BEFORE the custom code, so an upstream like http://host/api
produces /api/v1/storage/... after the join — then TrimPrefix("/v1")
is a no-op because the string starts with /api. Fix: strip /v1
BEFORE calling origDirector. New TestProxy_SubPathUpstream regression
locks this in. Today: bare-host URLs only, dormant — but moving
gateway behind a sub-path in prod would have silently 404'd.
O-WARN2: url.Parse is permissive — typo "127.0.0.1:3211" (no scheme)
parses fine, produces empty Host, every request 502s. mustParseUpstream
fail-fast at startup with a clear message naming the offending
config field.
Dismissed (3, all Kimi, same false TrimPrefix theory):
K-BLOCK "TrimPrefix loops forever on //v1storage" — false, single
check-and-trim, no loop
K-WARN "no upper bound on repeated // removal" — same false theory
K-WARN "goroutines leak if upstream parse fails while binaries
running" — confused scope; binaries are separate OS processes
launched by the smoke script
D1 smoke updated (post-D6): the 501 stub probes are gone (gateway no
longer stubs /v1/ingest and /v1/sql). Replaced with proxy probes that
verify gateway forwards malformed requests to ingestd and queryd. Launch
order changed from parallel to dep-ordered (storaged → catalogd →
ingestd → queryd → gateway) since catalogd's rehydrate now needs
storaged, queryd's initial Refresh needs catalogd.
All six G0 smokes (D1 through D6) PASS end-to-end after every fix
round. Phase G0 substrate is complete: 5 binaries, 6 routes, 25 fixes
applied across 6 days from cross-lineage review.
G1+ next: gRPC adapters, Lance/HNSW vector indices, Go MCP SDK port,
distillation rebuild, observer + Langfuse integration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
102 lines
3.4 KiB
Go
102 lines
3.4 KiB
Go
// gateway is the Lakehouse-Go HTTP ingress. D6 promotes the D1
|
|
// stub endpoints into real reverse-proxies fronting all four backing
|
|
// services (storaged, catalogd, ingestd, queryd) on a single bind.
|
|
//
|
|
// Routes:
|
|
// /v1/storage/* → storaged
|
|
// /v1/catalog/* → catalogd
|
|
// /v1/ingest → ingestd
|
|
// /v1/sql → queryd
|
|
//
|
|
// The /v1 prefix lives at the edge — internal services route on
|
|
// /storage, /catalog, /ingest, /sql. Per Kimi K2 finding from the
|
|
// D1 plan review: httputil.NewSingleHostReverseProxy preserves the
|
|
// inbound path by default, so the proxy helper strips /v1 in its
|
|
// Director before forwarding.
|
|
package main
|
|
|
|
import (
|
|
"flag"
|
|
"log/slog"
|
|
"net/url"
|
|
"os"
|
|
|
|
"github.com/go-chi/chi/v5"
|
|
|
|
"git.agentview.dev/profit/golangLAKEHOUSE/internal/gateway"
|
|
"git.agentview.dev/profit/golangLAKEHOUSE/internal/shared"
|
|
)
|
|
|
|
func main() {
|
|
configPath := flag.String("config", "lakehouse.toml", "path to TOML config")
|
|
flag.Parse()
|
|
|
|
cfg, err := shared.LoadConfig(*configPath)
|
|
if err != nil {
|
|
slog.Error("config", "err", err)
|
|
os.Exit(1)
|
|
}
|
|
|
|
upstreams := map[string]string{
|
|
"storaged_url": cfg.Gateway.StoragedURL,
|
|
"catalogd_url": cfg.Gateway.CatalogdURL,
|
|
"ingestd_url": cfg.Gateway.IngestdURL,
|
|
"queryd_url": cfg.Gateway.QuerydURL,
|
|
}
|
|
for k, v := range upstreams {
|
|
if v == "" {
|
|
slog.Error("config", "err", "gateway."+k+" is required")
|
|
os.Exit(1)
|
|
}
|
|
}
|
|
|
|
// Per scrum O-WARN2 (Opus): url.Parse is permissive — a typo
|
|
// like "127.0.0.1:3211" (missing scheme) parses without error
|
|
// but produces empty Host, and every proxied request 502s. Fail
|
|
// fast at startup if scheme/host are missing so misconfigs
|
|
// surface in `systemctl status gateway` rather than at first traffic.
|
|
storagedURL := mustParseUpstream("storaged_url", cfg.Gateway.StoragedURL)
|
|
catalogdURL := mustParseUpstream("catalogd_url", cfg.Gateway.CatalogdURL)
|
|
ingestdURL := mustParseUpstream("ingestd_url", cfg.Gateway.IngestdURL)
|
|
querydURL := mustParseUpstream("queryd_url", cfg.Gateway.QuerydURL)
|
|
|
|
storagedProxy := gateway.NewProxyHandler(storagedURL)
|
|
catalogdProxy := gateway.NewProxyHandler(catalogdURL)
|
|
ingestdProxy := gateway.NewProxyHandler(ingestdURL)
|
|
querydProxy := gateway.NewProxyHandler(querydURL)
|
|
|
|
if err := shared.Run("gateway", cfg.Gateway.Bind, func(r chi.Router) {
|
|
// Storage / catalog have multi-segment paths under their
|
|
// prefix (e.g. /v1/storage/get/<key>). chi's `*` wildcard
|
|
// captures the rest of the path.
|
|
r.Handle("/v1/storage/*", storagedProxy)
|
|
r.Handle("/v1/catalog/*", catalogdProxy)
|
|
// Ingest + sql are single endpoints. We accept any method
|
|
// (GET/POST/etc) and let the backing service decide. ingestd
|
|
// only accepts POST; queryd only accepts POST. Other methods
|
|
// will get the backend's 405.
|
|
r.Handle("/v1/ingest", ingestdProxy)
|
|
r.Handle("/v1/sql", querydProxy)
|
|
}); err != nil {
|
|
slog.Error("server", "err", err)
|
|
os.Exit(1)
|
|
}
|
|
}
|
|
|
|
// mustParseUpstream parses an upstream URL string and validates that
|
|
// scheme + host are non-empty. Exits the process on failure — gateway
|
|
// can't function without a valid upstream so failing fast is the
|
|
// right call. Per scrum O-WARN2.
|
|
func mustParseUpstream(name, raw string) *url.URL {
|
|
u, err := url.Parse(raw)
|
|
if err != nil {
|
|
slog.Error("config", "err", "parse "+name+": "+err.Error())
|
|
os.Exit(1)
|
|
}
|
|
if u.Scheme == "" || u.Host == "" {
|
|
slog.Error("config", "err", name+" must include scheme + host (got "+raw+")")
|
|
os.Exit(1)
|
|
}
|
|
return u
|
|
}
|