The 2026-05-01 persistent-stack milestone exposed two collision
modes between the long-running Go stack and the pre-push smoke
harness:
1. PKILL COLLISION: smoke teardown uses anchored
`pkill -f "bin/(storaged|...|gateway)$"`. Same-named persistent
processes match → smokes kill 7 of 11 persistent daemons.
2. MINIO STATE COLLISION: persistent stack writes
`_vectors/workers.lhv1` to the shared lakehouse-go-primary
bucket. Smoke vectord rehydrates from same bucket → sees both
smoke-owned and persistent-owned indexes → assertion failures.
Both fixed in this commit by adding two isolation layers:
LAYER 1 — distinct binary names via symlink:
bin/persistent-<daemon> → bin/<daemon>
Persistent stack runs as ./bin/persistent-gateway etc.
Smoke pattern `bin/(name)$` matches `bin/gateway$` but NOT
`bin/persistent-gateway$` (regex group requires bin/ followed
immediately by a daemon name; "bin/p..." doesn't qualify).
Cmdline lookup verified: 7 persistent procs, 0 match smoke pkill.
LAYER 2 — separate MinIO bucket via temp config:
Persistent stack writes to lakehouse-go-persistent (configurable
via $LH_PERSISTENT_BUCKET). Temp toml at /tmp/lakehouse-persistent.toml
inherits everything from lakehouse.toml except [s3].bucket which
is sed-replaced. Bucket auto-created via mc if missing.
Verified: workers.lhv1 lands in persistent bucket; primary
bucket _vectors/ stays empty.
Net effect: the persistent stack should survive `git push` (which
runs smokes that rehydrate vectord from primary bucket and pkill
their own bin/<name>$ daemons). This commit is the first push test
WITH the persistent stack live.
Caveat: bin/persistent-* symlinks are gitignored already (/bin/ is
in .gitignore wholesale), so the symlinks need to be created on
each fresh checkout — which start_go_stack.sh does idempotently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
c48b58ff8d