golangLAKEHOUSE/docs/CLAUDE_REFACTOR_GUARDRAILS.md

Claude Refactor Guardrails — Go Lakehouse

Mission

Continue the Go refactor without recreating the Rust-era complexity.

The Go rewrite exists to make the lakehouse operationally legible:

• small binaries
• clear service boundaries
• gateway-fronted APIs
• smoke-testable behavior
• fast build/run/debug loop
• no accidental framework bureaucracy

The Rust repo had maturity, but also accumulated control-plane weight: validator, auditor, provider routing, iteration loops, UI, truth layers, and agent-era scaffolding. Do not blindly port all of that back.

Prime Directive

Preserve the Go spine:

gateway
  → storaged
  → catalogd
  → ingestd
  → queryd
  → vectord
  → embedd
Only add complexity when there is measured evidence, a failing smoke, or a documented feature-parity requirement.

Refactor Rules
1. No silent behavior changes

Before changing any service behavior, identify:

current route
current request schema
current response schema
current status-code behavior
current smoke test covering it

If no smoke exists, add one before or with the refactor.

2. Keep service boundaries hard

Do not let services reach into each other’s internals.

Allowed:

HTTP client calls
shared request/response structs when stable
small internal packages for config/secrets/logging

Avoid:

importing another service’s implementation package
hidden global state
“just for now” shared mutable registries
circular service knowledge
3. Go is not Rust

Do not imitate Rust patterns mechanically.

Prefer Go-native clarity:

simple structs
explicit errors
small interfaces at the consumer side
context-aware HTTP handlers
table-driven tests
boring package names
no abstraction tax unless repeated 3+ times
4. Validation replaces the borrow checker

Rust caught many problems at compile time. Go will not.

Therefore every refactor must preserve or improve:

input validation
dimension checks
duplicate handling
restart persistence checks
schema drift detection
error status mapping
smoke coverage
5. Performance work must be measured

Do not optimize by vibes.

For each performance change, record:

baseline command
baseline result
changed code path
new result
regression risk
rollback plan

Current known bottleneck:

vectord Add is RWMutex-serialized.
500K vectors: ~35m36s, ~234/sec avg.
GPU around 65%, so embedding is not the only bottleneck.

Do not claim concurrency improvements unless the HNSW library thread-safety is audited or writes are safely batched/sharded.

File/Package Expectations
cmd/

One binary per service. Keep main files thin.

Main should only:

load config
construct dependencies
wire routes
start server
handle shutdown
internal/

Shared code belongs here only when it is genuinely shared.

Good internal packages:

config
secrets
storeclient
catalogclient
gateway routing helpers
logging
request/response contracts

Bad internal packages:

vague “utils”
giant “common”
cross-service god objects
hidden dependency containers
scripts/

Every major behavior needs a runnable smoke.

Smokes are not decoration. They are the replacement nervous system.

Existing smoke pattern must remain:

d1 skeleton/health/gateway
d2 storaged
d3 catalogd
d4 ingestd
d5 queryd
d6 full ingest/query
g1 vectord
g1p vectord persistence
g2 embed → vector add → search

New functionality needs a new smoke or an extension to the closest existing one.

Refactor Checklist

Before editing:

 Read README.md
 Read docs/PRD.md
 Read docs/SPEC.md
 Read docs/DECISIONS.md
 Read docs/PHASE_G0_KICKOFF.md
 Identify affected services
 Identify affected smokes

During editing:

 Keep public API stable unless explicitly changing it
 Keep errors explicit
 Keep logs useful but not noisy
 Avoid package sprawl
 Avoid premature generic interfaces
 Preserve restart behavior
 Preserve gateway-only acceptance path

After editing:

 Run go test ./...
 Run relevant smoke script
 Run full smoke loop when service contracts changed
 Record evidence in a short refactor note

Full smoke loop:

for s in scripts/{d1,d2,d3,d4,d5,d6,g1,g1p,g2}_smoke.sh; do
  "$s" || break
done
Refactor Note Format

Create or update:

docs/refactor-notes/YYYYMMDD-<short-name>.md

Use this structure:

# Refactor Note: <name>

## Goal

## Files changed

## Behavior changed

## Behavior preserved

## Tests run

## Smoke results

## Performance before/after

## Risks

## Rollback
Anti-Patterns To Reject
Reject these unless specifically requested:

porting Rust modules 1:1
adding orchestration before service parity
adding AI/agent logic inside core services
making gateway business-aware
hiding failures behind retries
swallowing errors
“temporary” global maps
changing route contracts without smoke updates
adding dependencies for trivial code
optimizing vector ingestion without measurement
rebuilding the Rust bureaucracy in Go clothing
Preferred Next Targets

Prioritize in this order:

Stability of existing Go service contracts
Better smoke coverage
Persistence limits and large object handling
vectord ingestion bottleneck analysis
gateway observability
feature parity with Rust only where needed
UI/agent/auditor layers later, not now
Architectural Position

The Go rewrite should remain the production spine.

The Rust system remains historical reference and possible source for:

validation ideas
audit semantics
provider-routing lessons
prior acceptance criteria
edge cases

But Rust is not the shape to copy.

Go owns the clean operational path.
Rust owns historical scar tissue and high-performance lessons.

Do not confuse the two.


Also add a shorter command prompt when you hand this to Claude Code:

```md
Read `docs/CLAUDE_REFACTOR_GUARDRAILS.md` first.

Then inspect the current Go lakehouse repo and produce a refactor plan only. Do not edit code yet.

Your plan must identify:
1. affected services
2. affected routes
3. affected request/response contracts
4. affected smoke scripts
5. risks of accidentally reintroducing Rust-era complexity
6. exact tests/smokes you will run after changes

Do not port Rust structure blindly. Preserve the Go service spine.